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ABSTRACT 


The  U.S.  Navy  employs  a  single  approach  to  education  and  training  in  virtually 
all  of  its  schoolhouses  and  learning  environments.  This  “one  size  fits  all”  system  is  dated 
and  inefficient,  and  the  Navy  could  potentially  benefit  from  an  individualized  approach. 
Personalized  learning  is  a  methodology  that  enables  the  individual  student  to  learn  in  a 
manner  that  best  suits  his  or  her  aptitude,  background,  and  learning  style.  This  approach, 
while  complex  and  expensive  to  implement,  is  quickly  gaining  traction  as  educational 
technology  improves.  The  benefits  of  such  a  methodology  to  student  outcomes  and 
organizational  efficiency  could  be  substantial. 

In  analyzing  cost  structures  of  three  fundamental  instructional  models,  long-run 
average  total  costs  for  each  were  found  to  be  most  sensitive  to  delivery  of  instruction,  not 
content  development  or  school  infrastructure.  Fewer  human  teachers,  less  travel  time, 
more  cost-effective  delivery  of  training,  and  a  higher  level  of  student  performance  make 
personalized  learning  an  attractive  alternative  to  the  industrial  model.  The  Navy’s  Digital 
Tutor  program  is  one  such  example,  and  although  there  are  mixed  results  for  its 
effectiveness  and  cost  savings,  evaluating  this  program  provides  lessons  for  continued 
efforts  in  embracing  technology  to  develop  revolutionary  training  and  education 
programs  for  the  future. 
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EXECUTIVE  SUMMARY 


As  opposition  to  the  “industrial-era”  approach  to  education  gains  popularity, 
several  emerging  pedagogies  and  novel  approaches  to  education  stand  poised  to  displace 
the  traditional,  one-size-fits-all  classroom  model  for  learning.  Personalized  learning  (PL) 
is  one  such  methodology,  which  seeks  to  enable  the  individual  student  to  learn  in  a 
manner  that  best  suits  his  or  her  aptitude,  background,  and  learning  style.  This  approach, 
while  potentially  complex  and  expensive  to  implement,  is  quickly  gaining  traction  as 
educational  technology  improves.  The  benefits  of  such  a  methodology  could  be 
substantial,  both  in  terms  of  student  outcomes  and  of  organizational  efficiency.  The  U.  S. 
Navy  currently  employs  a  single  approach  to  education  and  training  in  virtually  all  of  its 
schoolhouses  and  learning  environments.  This  “one  size  fits  all”  system  is  dated  and 
arguably  quite  inefficient  in  many  cases,  and  the  Navy  could  potentially  benefit  from  a 
more  individualized  approach  if  implemented  properly. 

Technology-enabled  individualized  instruction  can  be  more  effective  than 
traditional  models;  however,  effectiveness  is  only  half  of  the  equation.  The  upfront  costs 
of  developing  and  implementing  these  tools  are  likely  to  be  substantial,  if  not  altogether 
prohibitive.  In  order  to  fully  inform  decision  makers,  both  costs  and  benefits  of  programs 
must  be  presented.  Unfortunately  most  assessments  of  educational  solutions  tend  to  focus 
on  effectiveness,  or  potential  benefits,  without  recognizing  the  costs  (or  cost  savings) 
(Carey  et  al.,  2007).  We  want  our  decision  makers  to  efficiently  allocate  resources,  which 
implies  maximizing  effectiveness  for  any  given  cost  and  minimizing  cost  for  any  given 
benefit.  In  order  to  accomplish  this  goal,  they  must  understand  the  actual  value  of  training 
provided  by  technology-based  PL  solutions. 

A.  MOTIVATION 

The  education  enterprise  has  historically  lagged  other  industries  in  terms  of 
technology  adoption,  creating  an  ever  growing  disconnect  in  experiences  from  academic 
environment  to  the  workplace.  Additionally,  recent  claims  have  surfaced  that  the  current 
“millennial”  generation  not  only  has  a  drastically  different  relationship  with  technology, 
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but  also  fundamentally  different  ways  of  thinking  and  acquiring  knowledge  than  previous 
generations  (Prensky,  2001);  this  can  render  the  current  system  ineffective  for  learning. 
Rapid  advances  in  information  and  communication  technologies  (ICT)  are  fundamentally 
changing  the  nature  of  work  in  the  military,  with  cognitive  skills  becoming  increasingly 
important.  Also,  the  recruits  who  enter  the  service  are  more  and  more  accustomed  to  the 
integration  of  technology  with  their  daily  lives,  they  have  come  to  expect  the  same  in 
their  school  and  work  lives. 

B.  UNDERSTANDING  PERSONALIZED  LEARNING 

Bloom  (1984)  found  that  individually  tutored  students  performed  two  standard 
deviations  above  the  average  conventionally  taught  student.  In  other  words  the  average 
tutored  student  scored  higher  than  98  percent  of  the  control  group.  This  is  commonly 
known  as  “the  two-sigma  problem,”  and  the  aim  of  technology-based  personalized 
learning  is  to  reclaim  as  much  of  that  two  standard  deviation  advantage  as  possible,  not 
by  hiring  a  dedicated  teacher  for  every  student,  but  rather  by  leveraging  technology 
(Fletcher,  1992). 

In  a  PL  system,  varied  instructional  actions  are  implemented  based  on  student 
attributes  to  account  for  differences  between  learners.  The  observed  learner  differences 
that  are  typically  used  in  prescribing  PL  solutions  are  intellectual  abilities  and  prior 
knowledge,  cognitive  and  learning  styles,  and  motivation  and  other  personality  traits 
(Como  &  Snow,  1986).  Methods  of  accounting  for  these  differences  include  altering  or 
adjusting  pace  of  instruction,  difficulty  of  material,  sequence  of  content  presented, 
instructional  strategy,  and  instructional  delivery  method  (Wufleck,  2009). 

In  addition  to  improved  learning  outcomes,  PL  may  also  lead  to  greater 
efficiencies  and  cost  savings  in  the  training  process.  Evidence  has  shown  that  technology- 
based  learning  can  affordably  enable  PL  systems,  and  that  these  systems  are  not  only 
more  effective,  but  are  also  more  adaptable  and  more  flexible  than  traditional  models.  As 
a  result,  a  “Rule  of  Thirds”  can  be  used  to  estimate  the  effects  of  technology-based 
learning.  The  rule  states  that,  in  general,  these  applications  will  lower  the  cost  of 
instruction  by  approximately  one-third  and  either  time-to-train  will  be  reduced  by  one- 


third,  or  the  knowledge  and  skills  gained  will  increase  by  one-third  while  time-to-train 
remains  the  same  (Fletcher,  2010). 

Technology-based  PL  can  bring  educational  practices  into  the  21st  century, 
ensuring  that  students  are  well  prepared  for  the  new  challenges  they  will  face  in  a 
technology-heavy,  information  dependent  labor  market.  This  sentiment  applies  equally  to 
military  members  as  to  non-military  members,  if  not  more  so.  Our  military’s  competitive 
advantage  is  increasingly  tied  to  technological  dominance,  and  skills  required  of  our 
service  members  are  heavily  weighted  in  the  cognitive  domain  (Fletcher,  2010). 
Maintaining  operational  adaptability  in  complex  environments  is  paramount,  and  a 
system  of  learning  that  engages  the  active  learner,  encourages  critical  thinking,  and 
requires  demonstration  of  mastery  before  advancement  is  the  key  to  defeating  current  and 
future  adversaries  (Department  of  the  Army,  2011). 

C.  INTELLIGENT  TUTORING  SYSTEMS 

When  properly  implemented,  technological  tools  have  the  power  to  maximize 
student  potential,  capturing  the  benefits  of  one-to-one  tutoring  that  Bloom  (1984) 
articulated.  PL  technologies  that  are  responsive,  that  can  adjust  real-time  to  the  learner’s 
preferred  style  or  speed,  prior  knowledge,  and  aptitudes,  can  ensure  that  all  students  reach 
their  maximum  potential  with  no  time  wasted,  often  at  a  cost  savings  (Dede,  2013). 

Computer-assisted  instruction  programs  that  incorporate  elements  of  artificial 
intelligence  (AI)  and  machine  learning  to  facilitate  the  understanding  of  student  level  of 
knowledge,  response  to  individual  needs,  and  teaching  based  on  learning  style  or 
preferences,  all  in  real  time,  are  commonly  referred  to  as  Intelligent  Tutoring  Systems 
(ITS).  Today’s  ITS  can  understand  a  student’s  thought  processes,  motivations,  and 
emotions,  essentially  creating  a  mentor  for  every  learner.  Machine  learning  and  data 
mining  techniques  also  support  this  understanding  of  students  and  how  they  learn 
(Woolf  et  al.,  2013). 

Because  of  their  inherent  complexity  and  the  need  for  such  in-depth 
representation  of  the  knowledge,  development  time  and  costs  are  major  limitations  of 
ITS.  Estimates  as  high  as  200  man-hours  of  construction  time  per  one  hour  of  instruction 

xvii 


highlight  the  requirement  for  significant  upfront  investments  to  develop  an  ITS  (Fletcher, 
2010).  Clearly,  there  is  a  need  for  innovations  in  software  development  to  mitigate 
potentially  prohibitive  costs  of  constructing  ITS,  and  two  possible  solutions  to  the 
problem  are  common  authoring  tools  and  modularity. 

The  learner’s  progression  through  a  PL  program  should  be  competency-based, 
with  self-paced  advancement  occurring  only  after  demonstrating  mastery  of  the  material. 
This  progression  is  achieved  by  providing  instruction  in  a  manner  best  suited  to  both  the 
subject  matter  and  to  the  individual  preferences  of  the  learner.  Finally,  the  learning 
environment  should  be  malleable,  facilitating  important  social  interactions  between  peers 
and  with  the  mentor,  while  enabling  individual  progress  through  the  content. 

D.  COST  STUDIES 

Cost  studies  combine  an  objective  and  quantitative  determination  of  program 
effects  with  a  thorough  and  detailed  evaluation  of  costs,  based  on  sound  economic 
principles.  Benefits,  or  impacts  of  the  program,  are  compared  with  costs  through  a  variety 
of  methods  to  arrive  at  a  net  benefit  or  some  other  measure  of  value.  The  most  commonly 
used  types  of  cost  study  for  educational  evaluation  is  cost-effectiveness  analysis,  which 
compares  non-monetary  benefits  to  monetary  costs.  In  general,  a  program  is  worth 
adopting  if  it  demonstrates  greater  effectiveness  for  the  same  cost  than  the  alternatives  or 
status  quo,  the  same  effectiveness  at  lower  cost,  or  the  highest  cost-effectiveness  ratio 
(Ross  et  al.,  2007). 

Analyzing  the  relationship  between  cost  components,  such  as  fixed,  variable, 
average  total,  and  marginal  costs  provides  another  perspective  for  understanding  the 
differences  between  human-tutoring,  teacher-led  classrooms,  and  technology-enabled 
personalized  learning.  Economies  and  diseconomies  of  scale  become  more  apparent,  as 
do  the  differences  in  unit  costs  of  education.  Also,  we  can  better  illustrate  the  benefits 
provided  by  PL,  by  describing  each  student’s  effective  learning  level  as  a  marginal 
benefit.  This  basic  microeconomic  assessment  of  cost  curves  is  intended  to  serve  as  a 
theoretical  model  for  comparing  and  contrasting  the  cost  behavior  of  differing  approaches 
to  instruction. 
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All  three  instructional  models  point  out  the  critical  concept  that  long-run  average 
total  costs  are  most  sensitive  to  delivery  of  instruction,  not  content  development  or  school 
infrastructure.  Fewer  human  teachers,  less  travel  time  and  cost,  more  cost-effective 
delivery  of  training,  and  a  consistently  high-level  of  student  performance  make  PL  a 
seemingly  attractive  alternative  to  the  industrial  model,  although  candidate  programs 
should  be  analyzed  on  a  case-by-case  basis  to  determine  individual  viability  before 
conversion. 

E.  DIGITAL  TUTOR  CASE  STUDY 

The  U.  S.  Navy’s  Digital  Tutor  (DT)  program  is  an  ITS  currently  in  use  in  the 
initial  technical  training  phase  for  Sailors  in  the  Information  Systems  Technician  (IT) 
rating,  and  represents  one  of  the  most  advanced  PL  initiatives  being  undertaken  by  the 
military.  A  recent  comparison  of  DT-trained  Sailors  with  traditionally  trained  Sailors  has 
raised  some  questions  as  to  the  effectiveness  of  DT,  threatening  the  program’s  continued 
existence. 

To  determine  the  effectiveness  of  DT,  the  Institute  for  Defense  Analyses  (IDA) 
conducted  an  assessment  comparing  DT  graduates  to  graduates  of  the  traditional 
Information  Technology  Training  Continuum  (ITTC)  and  to  Fleet-experienced  ITs,  in  a 
series  of  exercises  meant  to  replicate  real-world  problems.  Overall,  the  DT  subjects 
drastically  outperformed  the  ITTC  and  Fleet  groups,  in  some  cases  by  a  substantial 
margin.  These  results  lead  the  authors  to  conclude  that  it  is  possible  to  develop  real 
expertise  in  a  drastically  accelerated  timeline  and  via  ITS  technical  training  only 
(Fletcher  &  Morrison,  2014). 

While  it  was  recognized  that  DT  was  quite  expensive  to  develop  and  implement, 
the  authors  concluded  that  the  cost  in  NPV  of  ITTC  was  62  percent  more  per  learner  than 
DT  training,  or  about  $180,000.  This  difference  would  account  for  additional  on-the-job 
training  required  by  ITTC  graduates  to  “catch  up”  with  the  DT  cohort  in  experience  level 
(Fletcher  &  Morrison,  2014).  If  this  result  is  taken  at  face  value,  then  perhaps  the  ROI 
justifies  a  high  initial  cost.  Herein  lies  part  of  the  problem  with  ITS  like  DT:  they  require 
a  significant  upfront  investment  and  a  longer  timeframe  to  realize  benefits. 


In  order  to  corroborate  IDA’s  findings  and  to  further  assess  the  cost-effectiveness 
of  the  DT  program,  the  Naval  Education  and  Training  Center  (NETC)  was  tasked  with 
completing  its  own  assessment  in  2016.  NETC’s  study  used  survey  data  of  recent  IT  A- 
school  graduates  and  their  supervisors,  designed  for  Kirkpatrick  level  three  evaluation  of 
training  transfer  to  job  performance.  The  study  compared  the  27-week  course  containing 
the  18-week  DT  module  to  the  37-week  traditional,  SYSADMIN  (SA)  course. 
Additionally,  cost  data  was  collected  and  analyzed  using  a  prototype,  Navy-developed 
costing  model.  The  findings  suggested  that,  at  best,  DT  was  equally  as  effective  as  SA, 
but  at  a  cost  of  $32,000  more  per  student.  However,  the  benefits  of  getting  a  Sailor  to  the 
fleet  10  weeks  sooner  with  DT  were  not  considered;  including  these  benefits  would 
reduce  the  cost  difference  and  enable  a  more  realistic  evaluation  of  the  program 
(Department  of  the  Navy,  2016). 

F.  CONCLUSION 

Findings  from  cost  studies  can  help  to  bolster  the  public’s  confidence  in  DOD’s 
management  of  taxpayer  dollars.  Without  such  information,  funding  agencies  cannot 
make  sound  decisions  regarding  implementation  of  new  education  and  training 
initiatives.  When  educational  evaluators,  not  economists,  consider  costs  of  programs, 
there  are  often  deficiencies  with  rigor,  quantity,  and  quality.  This  weakens  the  usefulness 
of  the  evaluations,  slows  development  of  understanding  of  what  is  truly  effective,  and 
slows  the  policy  making  process  (Ross  et  al.,  2007). 

Personalized  learning  approaches  enabled  by  technology,  such  as  ITS,  have  long 
demonstrated  value  as  effective  means  of  instruction,  but  little  attention  has  been  given  to 
their  overall  cost-effectiveness  (Angier  &  Fletcher,  1992).  Rapidly  advancing  techniques 
in  programming  and  AI,  along  with  increasingly  powerful  and  affordable  computer 
hardware  make  these  assessments  difficult.  Given  technology’s  impact  on  other 
industries  in  recent  past,  the  training  and  education  segment  should  follow  suit,  allowing 
new  tools  to  become  more  affordable  every  day.  While  the  costs  of  DT  may  yet  prove 
unsustainable  today,  we  should  not  be  discouraged  from  continuing  to  put  forth  effort  in 
embracing  technology  to  make  our  training  and  education  programs  second  to  none. 
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I.  INTRODUCTION 

Individualization  is  an  educational  imperative  and  an  economic  impossibility. 

— Michael  Scriven,  1975 

A.  BACKGROUND 

As  opposition  to  the  “industrial-era”  approach  to  education  gains  popularity, 
several  emerging  pedagogies  and  novel  approaches  to  education  stand  poised  to  displace 
the  traditional,  one-size-fits-all  classroom  model  for  learning.  Personalized  learning  is 
one  such  methodology,  which  seeks  to  enable  the  individual  student  to  learn  in  a  manner 
that  best  suits  his  or  her  aptitude,  background,  and  learning  style.  This  approach,  while 
potentially  complex  and  expensive  to  implement,  is  quickly  gaining  traction  as 
educational  technology  improves.  The  benefits  of  such  a  methodology  could  be 
substantial,  both  in  terms  of  student  outcomes  and  of  organizational  efficiency.  The  U.  S. 
Navy  currently  employs  a  universal  approach  to  education  and  training  in  virtually  all  of 
its  schoolhouses  and  learning  environments.  This  “one  size  fits  all”  system  is  dated  and 
arguably  quite  inefficient  in  many  cases,  and  could  potentially  benefit  from  a  more 
individualized  approach,  if  implemented  properly. 

In  this  study,  I  will  outline  both  how  and  why  individualized  instruction  can  be 
more  effective  than  traditional  models,  and  also  discuss  some  of  the  ways  that  technology 
can  enable  it;  however,  effectiveness  is  only  half  of  the  equation.  The  upfront  costs  of 
developing  and  implementing  these  tools  are  likely  to  be  substantial,  if  not  altogether 
prohibitive.  In  order  to  fully  inform  decision  makers,  both  costs  and  benefits  of  programs 
must  be  presented.  Unfortunately  most  assessments  of  educational  solutions  tend  to  focus 
on  effectiveness,  or  potential  benefits,  without  exploring  the  cost  side,  to  include  cost 
savings  (Carey  et  al.,  2007).  As  with  all  programmatic  decisions,  the  problem  is  one  of 
simple  economics.  We  want  our  decision  makers  to  efficiently  allocate  resources,  or 
maximize  effectiveness  while  minimizing  resources  expended.  In  order  to  accomplish 
this  goal,  they  must  understand  the  true  and  complete  value  of  training  provided  by 
technology-based  PL  solutions,  relative  to  the  costs  of  these  training  and  education 
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efforts.  This  is  even  more  important  in  an  era  of  fiscal  conservatism,  when  accountability 
is  paramount. 

B.  RESEARCH  QUESTIONS 

In  this  thesis,  I  will  attempt  to  answer  the  following  questions  regarding  the 
implementation  of  personalized  learning  in  Navy  training  and  education  settings: 

1.  How  might  personalized  learning  affect  student  outcomes  and 
organizational  efficiency  within  Navy  training  and  education 
environments? 

2.  How  can  current  and  future  educational  technology  enable  the  use  of 
personalized  learning  systems  within  the  Navy? 

3.  What  methods  of  cost  analysis  are  best  suited  to  evaluating  technology- 
enabled  personalized  learning  systems  for  Navy  use? 

C.  ORGANIZATION 

The  remainder  of  this  thesis  is  organized  into  five  chapters.  In  Chapter  II,  I  review 
the  literature  surrounding  learning  science,  generational  differences  in  learning,  and 
benefits  of  individualized  instruction.  Chapter  III  introduces  some  technology-based 
solutions  that  have  been  implemented  to  personalize  training  and  education,  including 
some  that  are  specific  to  DOD.  Chapter  IV  describes  the  types  of  cost  studies  used  to 
evaluate  educational  programs,  and  outlines  a  model  for  analyzing  cost  components  of 
alternative  courses  of  action.  Chapter  V  compares  two  studies  of  the  U.  S.  Navy’s  Digital 
Tutor  program,  pointing  out  strengths  and  weaknesses  of  each  and  recommending 
methods  for  improving  these  types  of  evaluations.  In  Chapter  VI,  I  make  some  final 
recommendations  for  further  study  and  use  of  personalized  learning  approaches,  as  well 
as  closing  remarks. 
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II.  LITERATURE  REVIEW 


Our  current  educational  norm  has  been  an  “industrialized,”  one-size-fits-all 
education  model  that  dates  back  to  the  19th  century  and  in  which  information  is  presented 
to  a  group  of  students  in  lecture  format  while  they  listen  passively  with  minimal  or  no 
participation.  This  type  of  education  is  a  far  cry  from  the  days  of  Aristotle,  when  the 
standard  was  one-to-one  tutoring  and  use  of  the  Socratic  method — critical  questioning  to 
allow  the  pupil  to  arrive  at  knowledge  on  his  or  her  own,  rather  than  having  it 
“transferred”  directly  from  the  teacher  (Murphy,  2006).  The  shift  over  the  generations  has 
had  primarily  economic  motivations,  particularly  in  the  United  States  during  the 
industrial  revolution,  when  Horace  Mann’s  common  school  model  for  mass  education 
gained  popularity  in  the  1830s  as  an  efficient  means  of  producing  future  factory  workers 
(Murphy,  2006). 

This  model  has  been  challenged  in  the  past  for  its  lack  of  effectiveness,  namely  by 
John  Dewey’s  progressive  movement  of  the  early  20th  century,  touting  a  child-centered 
pedagogy  (Apple  &  Teitelbaum,  2001),  Keller’s  personalized  system  of  instruction  (PSI) 
(1968),  and  Corno  and  Snow’s  adaptive  teaching  (1986).  More  recently,  a  wave  of 
initiatives,  studies,  and  new  policies  promote  a  learner-centric  environment  in  which 
instruction  can  be  tailored  to  the  student’s  past  experience,  competence,  learning  style, 
pace,  and  other  attributes.  These  alternative  methods  promise  to  reform  the  education 
system  by  producing  better  students  more  quickly,  and  students  who  are  more  prepared  to 
thrive  in  a  workplace  where  rapid  technological  advance  is  the  norm  and  critical  thinking 
skills  are  ever  more  in  demand. 

The  education  enterprise  has  historically  lagged  behind  other  industries  in  terms 
of  technology  adoption,  creating  an  ever-growing  disconnect  in  experiences  from 
academic  environment  to  the  workplace.  Additionally,  recent  claims  have  surfaced  that 
the  current  “millennial”  generation  not  only  has  a  drastically  different  relationship  with 
technology,  but  also  fundamentally  different  ways  of  thinking  and  acquiring  knowledge 
than  previous  generations  (Prensky,  2001),  thereby  rendering  the  current  system 

ineffective  for  learning  (Department  of  the  Air  Force,  2008). 
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In  an  often-cited  randomized  controlled  trial,  Bloom  (1984)  found  that 
individually  tutored  students  performed  much  better  than  those  taught  in  a  traditional 
classroom  format.  In  fact,  the  average  tutored  student  was  two  standard  deviations  above 
the  average  conventionally  taught  student,  as  shown  in  Figure  1.  In  other  words,  the 
average  tutored  student  scored  higher  than  98  percent  of  the  control  group.  This  is 
commonly  known  as  “the  two-sigma  problem,”  and  the  aim  of  personalized  learning  and 
other  methods  is  to  reclaim  as  much  of  that  two-standard-deviation  advantage  as 
possible — not  by  hiring  a  dedicated  teacher  for  every  student,  but  rather  by  leveraging 
technology  (Fletcher,  1992b). 


Achievement 


Figure  1.  Student  Achievement  Distribution  Based  on  Type  of  Instruction. 

Adapted  from  Fletcher  (2010). 


Two  primary  factors  are  believed  to  account  for  this  difference  in  outcomes.  The 
first  is  level  of  interactivity  between  teachers  and  students,  as  measured  by  the  number  of 
questions  a  teacher  and  students  may  ask  of  each  other  during  an  hour  of  instruction 
(Oblinger  &  Oblinger,  2005).  Graesser  and  Person  (1994)  found  that  students  were  only 
able  to  ask  about  .  1  questions  per  hour  in  a  classroom  setting,  compared  to  20-30  in  a 
tutored  session.  Teachers  asked  an  average  of  three  questions  per  hour  of  their  class, 
while  this  number  jumped  to  120-145  questions  asked  of  a  single  student  in  an  hour  of 
one-to-one  tutoring.  The  second  factor  is  the  tailoring  of  pace,  sequence,  and  content  to 
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the  students’  needs  (Fletcher,  2010).  A  tutor  is  able  to  fluidly  alter  speed  of  instruction, 
delivery  method,  or  content  based  on  continuous  assessment  of  their  students’  progress, 
easily  discerning  any  difficulties  along  the  way.  The  solutions  to  personalized  learning 
technology  I  present  in  this  paper  will  address  both  of  these  factors,  although  the  second 
is  more  aligned  with  the  overt  objectives  of  most  personalized  learning  systems. 

Today,  individualization,  personalization,  customization,  and  differentiation  are 
used  to  describe  teaching  and  learning  strategies  as  solutions  to  the  outdated,  one-size- 
fits-all  model  of  education.  Often  used  interchangeably,  there  is  no  agreed-upon 
definition  of  these  terms.  Typically,  however,  individualization  refers  to  pace  of 
instruction  alone,  while  differentiation  suggests  adjusting  the  learning  approach  based  on 
the  learner’s  interests,  abilities,  or  learning  style.  Personalization  can  be  thought  of  as  the 
combination  of  both  individualization  and  differentiation  (U.S.  Department  of  Education, 
2010;  Grant  &  Basye,  2014).  For  purposes  of  clarity,  I  will  use  the  term  “personalized 
learning”  (PL)  to  encompass  any  and  all  types  of  tailored  approaches  to  education, 
including  pace,  learning  style,  background  knowledge,  abilities,  preferences,  or  any  other 
attribute  which  may  be  used  to  adjust  to  the  individual.  These  alternatives  to  the  current 
classroom-learning  environment  seek  not  only  to  simply  modernize  outdated  mass 
education  models,  but  to  leverage  technology  to  transform  formal  education  into  an 
individualized,  learner-centered  endeavor  as  it  was  intended  more  than  2,000  years  ago, 
“an  Aristotle  for  every  Alexander”  (Fletcher,  2010). 

A.  CULTURAL  SHIFTS  AND  DRIVERS  FOR  CHANGE 

Rapid  advances  in  information  and  communication  technologies  (ICT)  are 
fundamentally  changing  the  nature  of  work  in  the  military,  with  cognitive  skills 
becoming  increasingly  important.  Also,  as  recmits  are  entering  the  service  more  and 
more  accustomed  to  the  integration  of  technology  with  their  daily  lives,  they  have  come 
to  expect  the  same  in  their  school  and  work  lives.  Technology  adoption  in  training  and 
education  has  historically  lagged  most  other  industries  (Domenech,  Sherman,  &  Brown, 
2016;  Oblinger  &  Oblinger,  2005).  Furthermore,  some  argue  that  the  millennial 
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generation’s  lifelong  exposure  to  ICT  has  made  the  current  education  system  ill  equipped 
to  properly  teach  them,  and  that  a  radical  generational  shift  in  learning  has  occurred. 

The  term,  “digital  natives,”  has  been  used  to  describe  members  of  the  current 
generation  born  after  1980.  Also  known  as  the  net  gen  or  millennials,  these  are 
individuals  who  have  never  known  a  world  without  the  Internet,  who  grew  up  connected 
and  surrounded  by  ICT  (Oblinger  &  Oblinger,  2005).  They  are  said  to  be  experts  in  the 
use  of  technology,  accustomed  to  receiving  their  information  digitally,  and  that  they  are 
more  active,  experiential  learners  rather  than  passive.  They  like  to  work  in  teams  and 
have  social  connections,  and  need  to  be  interactive,  whether  with  other  people  or  with 
their  technology  (Oblinger  &  Oblinger,  2005).  Some  have  argued  that  this  prolonged 
exposure  and  use  of  ICT  from  an  early  age  has  altered  the  way  that  the  entire  generation 
learns.  Prensky  (2001)  claimed  that  digital  natives  “think  and  process  information 
fundamentally  differently  from  their  predecessors.”  Presnky  and  supporters  believe  that 
the  education  system  must  evolve  to  accommodate  this  dramatic  change  in  students’ 
learning  styles  (Brown,  2005;  Roberts,  2005). 

The  “gamer  generation,”  also  digital  natives,  grew  up  playing  video  games,  a 
particularly  powerful  form  of  ICT  due  to  their  high  level  of  interactivity.  According  to 
Carstens  and  Beck  (2005),  playing  games  during  the  formative  years  of  early  childhood 
through  adolescence  has  “hard  wired”  them,  forming  neural  pathways  differently  than 
earlier  generations.  They  have  a  different  belief  system,  modeled  around  competition  and 
winning,  heavily  influenced  by  years  of  gaming.  Consequently,  new  tools  are  needed  to 
teach  this  new  generation,  and  one  recommendation  is  to  include  experience-based 
learning,  which  is  broken  up  into  small  units  and  self-paced  to  the  individual  learner,  in 
much  the  same  way  as  games  are  paced  to  the  player’s  level  of  skill.  They  also  need  more 
interactivity,  whether  with  people  or  technology.  These  traits  and  needs  are  consistent 
with  descriptions  of  digital  natives,  but  give  us  a  slightly  narrower  perspective. 

Although  these  claims  make  sense  intuitively,  several  studies  have  found  little 

scientific  basis  behind  them  (Bullen,  Morgan,  &  Qayyum,  2011;  Corrin,  Lockyer,  & 

Bennett,  2010;  Pedro,  2006).  Bennett,  Maton,  and  Kervin  (2008)  found  that  youth  use  of 

technology  was  lower  than  expected,  and  that  it  was  not  universal  across  the  generation, 
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but  varied  widely.  A  meta-analysis  by  Bullen  et  al.  (2011)  could  not  locate  any  empirical 
evidence  to  support  claims  that  the  millennial  generation  is  fundamentally  different  in 
behavioral  characteristics  or  learning  preferences.  Both  studies  found  that  the  primary 
factors  influencing  a  person’s  use  of  ICT  were  socio-economic  status,  ethnicity,  gender, 
age,  and  access  to  technology,  not  generational  membership.  For  the  most  part,  research 
has  shown  that,  on  an  individual  basis,  learning  styles  vary  widely  and  are  not 
generalizable  to  an  entire  generation.  Additionally,  learning  styles  change  over  time,  or 
are  situation  dependent,  making  the  claims  of  generational  specific  style  even  less 
credible  (Pedro,  2006).  More  information  on  learning  styles  and  their  relevance  to  PL  can 
be  found  later  in  this  chapter. 

Another  study  by  Corrin  et  al.  (2010)  of  those  born  after  1980  found  there  was 
wide  variation  in  how  students  rated  their  own  aptitude  with  technology.  Access  to 
technologies  such  as  laptop  computers  and  smart  phones  was  found  to  be  essentially 
ubiquitous  in  this  case,  so  although  most  everyone  had  access  to  the  technology,  their 
level  of  competence  was  not  homogenous.  Also,  subjects  tended  to  use  their  devices 
much  more  for  everyday  life  than  for  academic  purposes,  but  this  could  be  attributed  to 
lack  of  requirements  or  accommodations  in  the  academic  setting  as  much  as  the  students 
choosing  not  to  adopt  tech  for  use  in  school.  The  variety  of  aptitude,  access,  and  use  of 
ICT  in  and  out  of  school  found  in  this  study  show  that  the  digital  native  generalizations 
may  be  unfounded,  although  it  raises  the  question:  if  the  technology  were  available  and 
supported  in  schools,  would  students  even  want  to  use  it? 

Cengage  Learning  (2010)  conducted  a  survey  to  determine  how  technology-based 
learning  affects  student  engagement  and  learning  outcomes  in  higher  education.  The 
survey  showed  that,  although  70  percent  of  students  do  prefer  a  learning  environment  that 
uses  a  great  deal  of  technology,  there  must  be  adequate  support  for  students  to  use  the 
tech.  Teachers  assumed  that  students  were  more  tech-savvy  than  they  believed 
themselves  to  be,  and  students  felt  that  ICT  in  the  classroom  was  under  supported  by  the 
school  (Cengage  Learning,  2010;  Cengage  Learning  &  Eduventures,  2010).  Margaryan, 
Littlejohn,  and  Vojt  (2011)  found  no  evidence  to  show  that  students  were  generationally 
different  in  the  way  they  created  and  shared  knowledge,  and  that  their  preferences  for  use 
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of  ICT  in  the  classroom  was  actually  driven  more  by  their  instructors’  preferences  than 
any  other  factor.  The  students  tended  to  conform  to  both  the  pedagogy  and  technology 
level  that  the  teacher  dictated.  By  contrast.  Usher  (2012)  reported  that  the  addition  or  use 
of  technology,  such  as  games  and  other  interactive  multimedia  experiences  in  schools, 
could  have  positive  motivational  effects  on  students.  These  interactive  technologies, 
because  they  can  be  readily  modified,  adjust  to  a  student’s  individual  skill  level,  in 
addition  to  fostering  teamwork  and  communication.  The  net-gen  feels  that  a  core  attribute 
of  technology  is  that  it  is  customizable  and  able  to  adapt  to  their  needs  (Oblinger  & 
Oblinger,  2005).  While  education  in  general  can  certainly  benefit  from  technology- 
enabled  programs,  the  emphasis  should  be  more  about  increasing  effectiveness  for  the 
individual  learner,  rather  than  accommodating  a  generation  believed  to  be  fundamentally 
different. 

Constant  use  and  exposure  to  ICT  is  likely  to  change  how  anyone  communicates, 
learns,  and  socializes,  regardless  of  age  or  generation  (Pedro,  2006).  Oblinger  and 
Oblinger  (2005,  p.  2.9)  remarked,  “We  are  all  products  of  our  environment-and 
technology  is  an  increasingly  important  part  of  that  environment.”  However,  the  question 
remains,  does  education  need  to  change  because  the  students  are  somehow  profoundly 
different  than  what  the  system  was  designed  to  teach?  Every  generation  has  its  own  social 
norms  and  cultural  practices,  and  millennials  are  no  exception.  Interaction,  social  and 
with  technology,  seems  to  be  an  important  theme  for  net  gen  learners.  Students  perform 
better  and  retain  more  when  they  actively  participate  in  the  construction  of  their  own 
knowledge  (Oblinger  &  Oblinger,  2005).  This  is  supported  by  the  evidence,  which  has 
shown  that  interaction  with  instructors  is  much  higher  in  the  one-to-one  setting,  and 
virtually  non-existent  in  a  lecture-based  classroom  (Graesser  &  Person,  1994).  It  has  also 
been  shown  that  technology  use  is  greater  in  students’  everyday  lives  than  in  school 
(Margaryan  et  al.,  2011).  At  home,  they  are  actively  engaged  in  the  digital  world,  finding 
their  own  way  to  new  knowledge,  while  in  school,  they  are  typically  subjected  to  passive 
learning  devoid  of  context. 

The  problem  is  that  not  enough  schools  and  teachers  are  embracing  the  tools 
available,  while  the  rest  of  the  world  has  done  so  with  gusto.  Students  not  exposed  to 
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certain  types  and  uses  of  ICT  while  in  school  might  find  themselves  at  a  disadvantage 
when  entering  the  workforce.  Digital  literacy  and  familiarity,  by  nature  of  widespread 
technological  advance,  are  becoming  required  basic  skills  for  employees,  and  the 
education  system  must  understand  this.  The  Army  and  Air  Force  have  both  recognized 
this  imperative,  though  perhaps  placing  too  much  emphasis  on  the  presumed  generational 
shift  (Department  of  the  Army,  2011;  Department  of  the  Air  Force,  2008).  Important  to 
remember  is  that  the  majority  of  millennials,  along  with  many  Gen-X  and  Baby  Boomers, 
do  indeed  have  an  elevated  comfort  with  technology,  and  that  can  be  leveraged  to 
implement  PL  systems  that  tailor  instruction  to  the  student’s  prior  experience,  preferred 
delivery  method/pace,  and  aptitudes.  Using  ICT  to  make  training  and  education  more 
active,  social,  and  learner-centered  would  address  DOD  priorities  that  its  service 
members  be  more  adaptive,  creative,  and  able  to  think  critically  to  ensure  our  military’s 
dominance  in  the  21st  century  (Department  of  the  Army,  2011;  Department  of  the  Air 
Force,  2008;  Department  of  the  Navy,  2012). 

B.  LEARNING  THEORY  AND  PL  TREATMENTS 

In  this  section,  I  will  review  PL  systems  that  tailor  instruction  to  the  student  based 
on  one  or  more  individual  attributes.  In  a  PL  system,  varied  instructional  actions  are 
implemented  based  on  student  attributes  to  account  for  differences  between  learners.  I 
will  also  describe  some  of  the  most  common  differentiating  characteristics  for  PL  and 
corresponding  methods  of  adapting  instruction.  The  observed  learner  differences  that  are 
typically  used  in  prescribing  PL  solutions  are  intellectual  abilities  and  prior  knowledge, 
cognitive  and  learning  styles,  and  motivation  and  other  personality  traits  (Corno  &  Snow, 
1986).  Methods  of  accounting  for  these  differences  include  altering  or  adjusting  pace  of 
instruction,  difficulty  of  material,  sequence  of  content  presented,  instructional  strategy, 
and  instructional  delivery  method  (Wufleck,  2009).  These  attributes  and  prescriptions 
will  all  be  discussed  further,  but  first  a  brief  synopsis  of  learning  and  instructional  theory 
will  lay  the  foundation  for  further  analysis. 
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1.  Learning  Science 

At  the  turn  of  the  century,  industrial  age  education  was  intended  to  prepare  large 
numbers  of  future  factory  workers  to  perform  simple,  repetitive  tasks  through  a  process 
of  remember  and  repeat.  These  workers  were  far  more  likely  than  today’s  laborers  to 
remain  in  one  single  profession  for  the  entirety  of  their  working  lifetime  (Oblinger  & 
Oblinger,  2005).  This  “transmission  paradigm”  of  passive  learning  is  most  closely  linked 
with  behaviorism,  the  prevailing  theory  of  learning  and  psychology  through  the  1950s. 
Behaviorists,  such  as  John  Watson  and  B.F.  Skinner,  sought  to  maintain  scientific 
credibility  by  only  acknowledging  those  reactions  that  could  be  objectively  observed  and 
measured  (Tennyson,  2010).  Internal,  cognitive  processes  were  necessarily  ignored,  and 
therefore,  a  stimulus-response  reinforcement  model  of  learning  was  adopted.  This  method 
focused  on  inputs  to  the  environment,  or  rewards  and  punishments,  which  could  be  used 
to  produce  a  desired  learning  outcome  (Pritchard,  2014).  Critical  thinking  was  not 
prioritized,  as  knowing  took  precedence  over  understanding.  Simply  achieving  the 
desired  behavior  via  conditioning  and  reinforcement  was  considered  a  success. 

In  the  postindustrial  age  characterized  by  rapid  technological  advance,  critical 
thinking,  problem  solving,  and  persuasive  expression  gained  importance  in  educational 
circles.  These  “new”  skills  came  to  define  an  era  of  understanding,  rather  than  knowing, 
and  behaviorism  gave  way  to  constructivism  as  the  dominant  theory  of  learning 
(Tennyson,  2010).  The  constructivist  theory,  with  roots  in  cognitive  science,  espouses 
that  learners  ‘construct’  new  knowledge  by  taking  in  information  and  using  it  to  build  on 
to  their  pre-existing  knowledge,  understanding,  and  experiences  (Brown,  2005).  This  is 
nearly  a  direct  antithesis  to  the  transmission  paradigm  of  the  past,  and  emphasizes  that 
effective  learning  is  an  active,  not  a  passive  process.  Additionally,  context  is  important, 
both  from  the  student’s  reflection  on  prior  knowledge  and  from  the  realism  of  the 
educational  setting  (Domenech  et  al.,  2016).  Finally,  learning  is  a  social  and  collaborative 
process,  where  interaction  with  teams,  teachers,  experts  and  peers  are  critical  to 
constructing  new  knowledge.  Learner  engagement  at  every  level  and  control  over  the 
process  are  key  requirements  for  a  constructivist  approach  (Brown,  2005). 
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Constructivism  has  continued  to  dominate  the  field  of  learning  science  and  has 
heavily  influenced  instructional  theory  and  instructional  systems  design  over  the  past  30 
years.  Designing  learner-centric  systems  that  account  for  and  interact  with  a  student’s 
prior  knowledge  and  experience  have  become  more  and  more  common.  However,  aspects 
of  behaviorism  have  proven  viability  in  certain  situations,  and  integrated  instructional 
design  theories  attempt  to  synthesize  aspects  from  multiple  educational  models  to 
accommodate  an  individual’s  learner’s  progress  at  achieving  a  specific  desired  learning 
outcome.  Technology  has  enabled  the  advance  of  these  integrated  theories  with  the 
creation  of  interactive,  intelligent  learning  systems.  These  flexible,  adaptable  programs 
will  be  discussed  in  Chapter  III,  and  further  support  the  constructivist  position  of  active 
learning  in  a  responsive  and  interactive  environment,  built  upon  the  individual’s  past 
knowledge  and  experience  (Tennyson,  2010). 

2.  Aptitude  Variables 

Corno  and  Snow  (1986)  define  three  broad  categories  of  aptitude  variables  related 
to  educational  performance:  cognition,  conation,  and  affection.  These  variables  are  the 
individual  characteristics  to  which  instruction  can  be  tailored  in  PL  systems.  Cognition, 
or  the  process  of  acquiring  knowledge  (Pritchard,  2014),  includes  intellectual  abilities 
and  pre-existing  knowledge.  Conation,  meaning  natural  tendencies,  covers  cognitive  and 
learning  styles,  and  affection,  referring  to  feelings  and  emotions,  includes  motivation  and 
other  personality  traits.  These  variables  combine  in  complex  mixtures  unique  to  every 
person  to  account  for  one’s  performance  in  a  given  learning  situation.  Another  concept  of 
educational  performance  is  Gardner’s  multiple  intelligence  theory,  which  says  that 
intelligence  is  based  upon  a  mix  of  aptitudes  across  nine  different  intellectual  categories: 
linguistic,  logical/mathematical,  musical,  spatial/visual,  kinesthetic,  interpersonal, 
intrapersonal,  naturalistic,  and  existential  (Pritchard,  2014).  These  traits  are  closely  linked 
to  modalities  of  learning,  and  will  be  discussed  along  with  cognitive  and  learning  styles. 
The  majority  of  research  has  focused  around  understanding  this  category,  and  how  these 
differences  might  be  leveraged  to  gain  positive  effects  on  individuals’  learning  outcomes. 
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a. 


Cognition  and  Learning  Styles 


Cognitive  and  learning  styles  are  often  conflated  and  mistakenly  used 
interchangeably,  but  the  terms  are  not  synonymous.  Cognitive  styles  are  higher-order 
classes  that  refer  to  the  way  that  a  person  typically  processes  information  and  solves 
problems,  and  are  considered  to  be  stable  over  the  long  term  (Holden,  201 1).  The  primary 
styles  that  have  been  identified  are  field-independent  and  field-dependent.  Field- 
independent  style,  briefly,  describes  detail-oriented  individuals  who  are  analytical, 
rational  thinkers  that  focus  at  the  task  level.  Conversely,  field-dependent  individuals 
think  holistically  and  approach  problems  from  a  more  global  perspective.  They  see  the 
bigger  picture,  but  may  lose  sight  of  the  finer  details  (Pai,  Adler,  &  Shadiow,  2006). 
Though  not  specifically  descriptive  of  a  learning  situation,  cognitive  styles  do  inform  and 
influence  the  individual’s  learning  style,  or  the  preferred/most  effective  way  a  person 
processes  information  to  create  knowledge. 

Learning  styles  have  been  recently  popularized  in  the  literature  as  the  primary 
means  by  which  to  personalize  learning.  Proponents  contend  that  matching  teaching 
styles  to  the  individual  learning  styles  of  students  is  the  surest  way  to  solve  the  two-sigma 
problem.  Others  claim  that  this  could  be  limiting,  and  propose  that  a  student  or  teacher’s 
awareness  of  their  own  learning  styles  enable  them  to  be  more  effective  in  the  present- 
day  classroom.  This  point  comes  from  the  concept  of  metacognition,  or  thinking  about 
one’s  own  way  of  thinking  (Pritchard,  2014).  With  knowledge  about  how  one’s  own 
mind  works,  particularly  in  an  academic  environment,  that  individual  is  able  to  develop 
an  approach  to  learning  that  works  best  for  them.  Metacognition  allows  a  deeper  level  of 
engagement  with  the  learning  process,  motivating  and  enabling  a  student  on  the  path  of 
lifelong  learning  (Manacapilli  et  al.,  2011). 

In  order  to  better  understand  learning  styles,  Coffield,  Moseley,  &  Hall  (2004) 
conducted  a  meta-analysis  of  the  research  from  this  complex  and  widespread  field.  They 
reviewed  71  learning-styles  models  for  reliability,  validity,  and  effectiveness  by 
consolidating  the  results  of  thousands  of  professional  papers  and  reports,  and  directly 
referencing  631  sources.  The  group  divided  the  original  71  models  into  five  families  of 

learning  styles,  based  on  their  underlying  theories.  Thirteen  representative  models, 

12 


deemed  to  be  the  most  significant  of  the  entire  group,  were  then  chosen  to  review  in- 
depth.  The  findings  from  this  meta-analysis  suggest  that  the  field  of  learning  styles 
research  is  not  at  all  consolidated,  and  at  times  contradictory  and  contentious.  Even  the 
final  group  of  13  could  not  be  further  merged  into  a  single,  coherent  theory  on  how 
people  differ  in  their  approaches,  characteristics,  and  tendencies  towards  learning.  While 
this  result  isn’t  conclusive,  the  report  does  find  sound  evidence  that  there  are  many 
legitimate  differences  among  learners,  and  that  further  study  and  agreement  on  these 
differences  could  lead  to  benefits  in  the  future  of  education  and  learning. 

One  popular  subset  of  learning  styles  is  modalities,  which  delineates  the 
individual’s  preference  for  sensory  input  of  information.  These  input  channels  are  visual, 
auditory,  kinesthetic  and  tactile,  and  are  typically  combined  in  some  form  of  multi- 
sensory  processing  as  a  preference  for  taking  in  information.  This  model,  typically 
referred  to  as  VAKT,  has  been  used  in  many  cases  to  personalize  e-leaming  systems  to 
individual  learners’  pedagogical  needs.  A  review  of  one  such  platform  by  Peter,  Bacon, 
and  Dastbaz  (2010)  found  potential  benefits,  but  agreed  that  the  field  is  too  dispersed  and 
that  not  enough  data  exists  as  to  which  learning  style  model  is  the  best,  or  most  effective 
for  these  applications.  A  study  conducted  by  Becker,  Kehoe,  and  Tennent  (2007),  also 
using  a  VAKT  model,  concluded  that  a  student’s  learning  style  did  not  seem  to  affect 
their  preference  for  alternative  delivery  methods  of  instructional  content.  However,  they 
did  find  that  traditional  methods  of  classroom  instruction  tend  to  be  favored  only  by  those 
with  learning  styles  that  match,  like  aural.  There  are  many  more  examples  of  research 
since  the  1960s  on  learning  styles  that  have  produced  similar,  inconclusive  results, 
leading  Coffield  et  al.  (2004),  Tobias  (1989),  Holden  (2011),  and  others  to  conclude  that 
the  benefits  of  personalizing  instruction  to  individual  learning  styles  have  been 
exaggerated  at  best,  and  at  worst,  is  a  wholly  ineffective  practice.  In  a  critical  and 
thorough  review  of  learning  styles  research,  cognitive  psychologists,  Pashler,  McDaniel, 
Rohrer,  and  Bjork  (2009,  p.  105)  found  “no  adequate  evidence  base  to  justify 
incorporating  learning  styles  assessments  into  general  educational  practice.”  All  agree 
that  much  more  concentrated  inquiry  is  needed  in  this  realm,  and  recommend  that  the 
research  methods  also  be  carefully  considered  beforehand.  Additionally,  there  may  be 
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evidence  to  show  that  educators  can  optimize  learning  outcomes  simply  by  ensuring  the 
delivery  mode  is  consistent  with  the  content  and  overall  learning  objectives,  without 
regard  to  individual  student  differences  (Hays,  2006).  This  has  been  also  linked  to  a 
multi-modal  approach  to  instruction,  in  which  delivery  mode  is  varied  throughout  the 
course  of  instruction,  ensuring  that  students  with  different  preferences  will  be  exposed  to 
their  optimal  style  enough  to  adequately  engage  with  and  grasp  concepts  being  taught 
(Pritchard,  2014). 

b.  Prior  Knowledge  and  Learning  Speed 

In  addition  to  conation  and  learning  styles,  individuals  express  differences  in 
cognition,  including  intellectual  abilities  and  existing  knowledge,  which  may  be  used  in 
PL  settings  (Corno  &  Snow,  1986).  While  intellectual  ability,  such  as  types  of 
intelligence,  is  a  complicated  and  not  well-understood  area,  prior  knowledge  does  show 
great  promise  for  explaining  differences  in  learner  outcomes  (Snow,  1986).  Past 
experience,  according  to  Tobias  (1989),  may  account  for  much  of  the  disparity  in  speed 
of  learning  among  students,  and  is  a  critical  variable  in  how  a  person  gains  new 
knowledge.  The  gap  in  learning  speed  has  been  shown  to  reach  ratios  of  more  than  4:1 
between  fastest  and  slowest  students,  highlighting  the  importance  of  adjusting  pace  of 
instruction  to  the  individual  (Fletcher,  2010).  Students  with  a  low  level  of  applicable 
prior  knowledge  simply  require  more  instructional  support  and  time  to  understand  new 
concepts. 

PL  systems  that  adjust  pace  of  instruction  may  lead  to  overall  better  learning 
outcomes  for  students,  in  addition  to  time  and  cost  savings  mentioned  earlier  with  the 
Rule  of  Thirds.  A  pioneer  in  individualized  education,  Keller  (1968)  created  his 
personalized  system  of  instruction  (PSI)  as  an  alternative  to  conventional  approaches  to 
teaching,  to  allow  students  to  work  completely  at  their  own  pace  through  a  course  of 
study  with  set  term  length.  Before  advancing  to  new  material,  they  were  required  to 
demonstrate  mastery  of  the  current  module.  Although  lectures  were  used  sparingly,  and 
only  as  a  source  of  motivation,  students  did  have  many  more,  less  formal  interactions 
with  instructors  and  facilitators  than  with  conventional  teaching,  keeping  engagement  at  a 
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high  level.  A  student  would  be  assessed  nearly  continuously  throughout  the  term,  but  as 
each  assessment  carried  a  very  low  weight,  it  was  not  considered  a  stressful  ordeal, 
thereby  possibly  interfering  with  learning.  In  Keller’s  study,  and  in  many  since,  students 
felt  a  greater  sense  of  accomplishment,  autonomy,  and  satisfaction  from  the  PSI  approach 
than  they  were  accustomed  to  in  traditional  classes.  Additionally,  an  improved  grade 
distribution  over  the  control  group  showed  that  the  students  had  learned  the  course 
material  more  thoroughly  than  their  traditionally  taught  counterparts.  While  popularity  of 
PSI  approach  waned  in  the  1990s,  modem  advances  in  educational  technology  such  as  e 
learning  have  revitalized  inquiry  and  interest  into  this  and  other  models  of  self-paced 
learning  (Eyre,  2007;  Hambleton,  Foster,  &  Richardson,  1998). 

In  a  more  recent  study  on  the  effectiveness  of  self-paced  learning,  Tullis  and 
Benjamin  (2011)  found  that  students  who  had  control  over  allocation  of  time  spent 
studying  particular  material  significantly  outperformed  those  who  had  no  control.  The 
self-paced  students  were  able  to  spend  less  time  on  topics  that  they  felt  were  easier  and 
more  time  on  those  they  found  to  be  more  difficult,  while  still  completing  the  overall 
course  in  the  same  amount  of  time.  This  success,  however,  was  believed  to  be  contingent 
on  the  students  using  an  appropriate  allotment  strategy,  likely  attained  through 
metacognition.  This  study  reiterates  the  importance  of  the  individual’s  understanding  of 
how  he  or  she  thinks  and  acquires  knowledge  and  understanding,  and  makes  the  case  for 
increased  emphasis  on  metacognition  in  a  constructivist  educational  environment 
(Pritchard,  2014). 

The  previous  two  examples  held  overall  course  time  constant,  while  allowing  the 
individual  to  progress  through  the  material  at  their  own  pace,  according  to  aptitude  and 
prior  knowledge.  All  students  would  still  finish  the  course  at  the  same  time,  although 
having  spent  individually  varying  amounts  of  time  on  course  concepts  or  modules.  At  the 
Center  for  Naval  Analyses,  Carey,  Reese,  Lopez,  Shuford,  &  Wills  (2007)  studied 
computer-based  self-paced  courses  in  Navy  initial  technical  training  (A-School) 
programs  to  determine  their  effect  on  overall  time  to  train.  Their  findings  revealed  a  10  to 
30  percent  reduction  in  training  time,  for  which  implications  of  cost  savings  are 
significant.  This  result  is  also  consistent  with  Fletcher’s  (2010)  Rule  of  Thirds  regarding 
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reduced  training  time,  reduced  cost,  and  improved  outcome  for  technology-based 
learning.  According  to  Fletcher,  time  savings  is  the  most  consistent  positive  result  of 
these  types  of  individualized  approaches  to  learning. 

c.  Affection  and  Learning 

The  third  category  of  Corno  and  Snow’s  (1986)  aptitude  variables  is  affection, 
which  covers  attributes  such  as  achievement  motivation,  engagement,  and  other 
personality  characteristics.  These  variables  tend  to  have  a  direct  impact  on  a  student’s 
level  of  effort  and  ultimately,  academic  achievement.  The  Center  on  Education  Policy,  in 
a  review  of  several  types  of  technology-based  educational  media,  such  as  video  games 
and  interactive  and  social  technology  tools,  found  that  students’  engagement  with  content 
and  motivation  to  learn  increased  when  these  tools  were  introduced  to  the  classroom 
(Usher,  2012).  Hattie  (1999)  demonstrated  in  a  meta-analysis  of  educational 
interventions,  that  a  student’s  disposition  toward  learning,  or  academic  motivation,  did 
indeed  have  a  significant  effect  on  student  achievement,  at  .61  standard  deviations  above 
the  mean,  whereas  a  .5  standard  deviation  improvement  would  approximate  one  letter 
grade  higher  on  an  exam.  While  the  underlying  mechanisms  for  changes  in  human 
affective  states  may  not  be  fully  understood  or  agreed  upon  by  the  academic  community, 
it  is  clear  that  emotion  does  play  an  important  part  in  the  mixture  of  aptitude  variables 
and  their  prediction  of  learner  achievement  (Cole,  Harris,  &  Field,  2004;  Corno  & 
Snow,  1986). 

C.  BENEFITS  OF  PERSONALIZED  LEARNING 

1.  Cost  Savings 

In  addition  to  vastly  improved  learning  outcomes,  PL  may  also  lead  to  greater 

efficiencies  and  cost  savings  in  the  training  process.  Evidence  has  shown  that  technology- 

based  learning  can  affordably  enable  PL  systems,  and  that  these  systems  are  not  only 

more  effective,  but  also  more  adaptable  and  more  flexible  than  traditional  models.  As  a 

result,  a  “Rule  of  Thirds”  can  be  used  to  estimate  the  effects  of  technology-based 

learning.  The  rule  states  that,  in  general,  these  applications  will  lower  the  cost  of 

instruction  by  approximately  one-third,  and,  either  time-to-train  will  be  reduced  by  one- 
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third,  or  the  knowledge  and  skills  gained  will  increase  by  one-third,  while  time  remains 
the  same  (Fletcher,  2010). 

This  finding  has  enormous  implications  for  an  organization  the  size  of  DOD, 
whose  success  absolutely  depends  on  training  and  education,  both  in  the  classroom  and  in 
the  field,  where  maintaining  operational  readiness  and  developing  leaders  and  tactical 
experts  is  an  ongoing  priority.  As  such,  the  costs  of  dedicated  “schoolhouse”  training  and 
education  among  the  services  account  for  approximately  $8.7  billion  per  year 
(Department  of  Defense  [DOD],  2015).  This  number  is  said  to  rise  to  upwards  of  $50B 
per  year  when  all  reservists,  DOD  civilians,  and  dependents  are  considered,  along  with 
other  learning  activities  not  specifically  accounted  for  in  the  Operation  &  Maintenance 
(O&M)  budget  (Fletcher,  2011).  In  recent  years,  fiscal  austerity  has  caused  increased 
pressure  to  find  ways  to  save  on  these  vital,  yet  expensive  requirements.  For  example, 
RAND  Corporation  studied  PL  (the  Air  Force  calls  it  customized  learning)  opportunities 
for  the  Air  Force  with  the  sole  intent  of  reducing  time-to-train,  and  thereby  significantly 
reducing  costs  (Manacapilli,  O’Connell,  &  Benard,  2011).  An  improvement  in  the 
learning  enterprise  as  a  whole  could  free  up  additional  resources  to  invest  in  operations, 
modernization  of  equipment,  or  other  programs  vital  to  national  defense. 

2.  Modernized  Skillset 

Another  potential  benefit  of  technology-based  PL  is  a  workforce  better  equipped 
to  thrive  in  the  technology-rich  world  of  today.  It  must  be  recognized  that  skills  required 
of  the  prototypical  worker  have  evolved  over  time.  In  the  transition  from  an  industrial 
society,  to  a  knowledge  based  one,  cognitive  skills,  such  as  critical  thinking,  problem 
solving,  creativity,  and  interpersonal  communication  have  widely  displaced  requirements 
for  manual  labor  (Woolf,  Lane,  Chaudhri,  &  Kolodner,  2013).  Additionally,  the  rapid 
advance  of  technology  necessitates  a  new  kind  of  agile  thinking  and  adaptability  that  the 
current  system  of  education  was  not  designed  to  address.  Many  learners  today  must 
endure  passive,  lecture-based  instruction,  absent  of  collaboration,  critical  thinking,  and 
deep  understanding.  Automation  of  physical  labor  and  proliferation  of  ICT  will  lead  to 
structural  unemployment  for  those  not  equipped  to  adapt  their  skillsets  to  new  vocations. 
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Technology-based  PL  can  bring  educational  practices  into  the  21st  century, 
ensuring  that  students  are  well  prepared  for  the  new  challenges  they  will  face  in  a 
technology-heavy,  information  dependent  labor  market.  This  sentiment  applies  equally  to 
military  members,  if  not  more  so.  Our  military’s  competitive  advantage  is  increasingly 
tied  to  technological  dominance,  and  skills  required  of  our  service  members  are  heavily 
weighted  in  the  cognitive  domain  (Fletcher,  2010).  Maintaining  operational  adaptability 
in  complex  environments  is  paramount,  and  a  system  of  learning  that  engages  the  active 
learner,  encourages  critical  thinking,  and  requires  demonstration  of  mastery  before 
advancement  is  the  key  to  defeating  current  and  future  adversaries  (Department  of  the 
Army,  2011). 

D.  CASE  STUDIES 

1.  Teach  to  One 

The  School  of  One  program  was  initiated  in  2009  in  New  York  City  middle 
school,  and  in  2012,  renamed  Teach  to  One,  and  further  expanded  to  include  15  schools 
in  several  urban  districts  on  the  east  coast  (U.S.  Department  of  Education,  2010).  The 
program  used  varying  technology-based  delivery  modes,  daily  assessments,  and  self- 
paced  learning  to  teach  math  to  nearly  6,000  academically  and  demographic  ally 
underprivileged  students  in  grades  five  through  eight.  A  lesson  bank,  containing  over 
1,000  lessons,  could  be  accessed  in  a  tailored  manner  to  learn  77  common  math  skills. 
Students’  results  on  a  standardized  test,  taken  at  the  beginning  and  end  of  the  school  year, 
were  compared  to  the  national  average  to  measure  effectiveness  of  the  program. 
Although  the  test  group  students  began  the  school  year  significantly  behind  the  national 
average,  roughly  .50  standard  deviations,  by  the  end  of  the  second  full  year  of  operation, 
students  had  achieved  results  of  .37  standard  deviations  above  the  national  mean.  In  other 
words,  Teach  to  One  students  showed  improvements  over  the  school  year  at  a  rate  47 
percent  higher  than  the  national  average  gains  (Ready,  2014).  Using  student  inputs  to 
establish  preferred  delivery  mode,  continuous  assessments  to  determine  level  of 
competency,  and  self-pacing,  the  Teach  to  One  program  is  an  encouraging  success  story 
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for  PL  in  a  K-12  setting,  and  is  currently  expanding  to  more  schools  nationwide  for 
further  development  and  testing. 

2.  Continued  Progress 

In  a  similar  study,  RAND  Corporation,  in  partnership  with  the  Bill  and  Melinda 
Gates  Foundation,  evaluated  62  K-12  schools  that  had  implemented  a  wide  variety  of  PL 
approaches  (Pane,  Steiner,  Baird,  &  Hamilton,  2015).  While  no  standard  methods  of  PL 
were  imposed  on  the  schools,  a  framework  of  five  strategies  was  common  across  the 
group:  learner  profiles  to  understand  individual  differences,  personal  learning  paths  to 
allow  student  control,  competency-based  progression  for  self-paced  achievement,  flexible 
learning  environments  to  enable  a  multi-modal  instruction,  and  emphasis  on  college  and 
workforce  preparedness.  The  overall  effect  size  found  across  the  62  study  schools  was 
.19  in  reading  and  .27  in  math.  While  this  is  a  statistically  positive  result,  it  falls  short  of 
Hattie’s  (1999)  average  intervention  effect  of  .40  standard  deviations,  suggesting  that 
there  are  other,  more  effective  approaches  that  might  have  been  implemented.  Of  course, 
until  the  costs  of  all  such  programs  are  weighed  along  with  these  benefits,  we  must 
reserve  judgment  as  to  their  relative  values.  Additionally,  because  the  PL  practices  varied 
so  widely  across  the  schools,  it  is  impossible  to  determine  whether  any  specific  approach 
types  or  particular  use  of  technology  had  a  more  positive  or  negative  effect,  potentially 
skewing  the  overall  result.  One  important  qualitative  observation  from  the  study  was  the 
general  lack  of  adoption  of  true  self-pacing  and  competency-based  advancement,  in  part 
due  to  pressure  to  maintain  traditional  grade-level  structure  and  content  in  order  to  meet 
externally  controlled  standardized  test  requirements  (Pane  et  al.,  2015).  This  statement 
highlights  outdated  policies  and  organizational  norms,  which  pose  major  barriers  to 
implementing  effective  PL  systems  in  any  traditional  educational  environment. 

3.  Project  RED 

Project  RED  was  a  large-scale  research  effort  across  997  K-12  schools  in  49 
states  that  began  in  2010  with  the  aim  of  discovering  how  educational  technology, 
particularly  computer  access  for  every  student,  might  improve  learning  outcomes  while 
remaining  cost  effective  (Greaves,  Hayes,  Wilson,  Gielniak,  &  Peterson,  2012).  The 

19 


study  found  that  technology,  when  properly  implemented,  improved  learning  outcomes 
while  saving  money  for  the  schools.  High-stakes  test  scores  and  graduation  rates  were 
significantly  higher,  while  dropout  rates  and  disciplinary  actions  were  lower  when  the 
ratio  of  students  to  computers  was  1:1.  The  greatest  returns  were  realized  when  the 
technology  was  fully  integrated  into  curricula  and  used  on  a  daily  basis.  The  authors  also 
found  that  school  administrators’  leadership  and  support  were  critical  to  the  success  of 
any  implementation  effort.  Although  this  program  did  not  claim  PL  as  its  primary 
objective,  the  authors  did  feel  that  individualized  instruction  was  the  most  significant 
reason  for  integrating  technology  into  education: 

Individualized  instruction  is  perhaps  the  most  important  use  model  of 
technology  in  education.  Whether  advanced  or  remedial,  individualized 
instruction  allows  students  to  learn  at  their  own  pace  and  engage  in 
learning  at  exactly  the  right  entry  point.  Technology-based  learning 
solutions  provide  almost  limitless  opportunities  for  personalization.  If  one 
approach  is  not  working  for  a  student,  alternatives  can  easily  be  tried  that 
are  better  suited  to  a  student’s  individual  learning  style  or  experiences. 
Because  students  are  in  active  control  of  their  learning,  they  are  more 
likely  to  stay  on  task.  (Greaves  et  al.,  2012,  p.  16) 

Though  Project  RED,  and  the  other  similar  programs  discussed  have  been  focused  on  K- 
12  education,  there  is  considerable  evidence  that  PL  does  have  potential  meaningful 
benefits.  If  properly  implemented  with  the  right  technology  and  with  buy-in  at  all  levels, 
PL  has  the  power  to  transform  the  notion  of  education,  as  we  know  it. 

E.  CHAPTER  SUMMARY 

Individualized  approaches  to  teaching  and  learning  can  be  traced  to  the  very 
origins  of  formalized  education,  more  than  two  thousand  years  ago.  Though  it  has  long 
been  understood  that  a  one-on-one  relationship  between  teacher  and  pupil  is  optimal  for 
learning  success,  the  model  is  generally  not  scalable,  and  has  succumbed  over  time  to 
more  cost-efficient  methodologies.  As  technology  advances  and  scientific  understanding 
of  how  we  think  and  learn  has  grown,  so  has  an  interest  in  reviving  the  concept  of  PL. 
Technology-enabled  PL  promises  not  only  to  improve  learning  outcomes  by  matching 
teaching  strategies,  methods,  and  timing  to  individual  student  differences,  but  it  seeks  to 
do  so  more  affordably  than  traditional  classroom  learning.  Thus  far,  the  evidence  is 
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mixed,  but  efforts  to  empirically  show  that  PL  is  more  effective,  and  that  it  can  capture 
the  two-sigma  difference  made  famous  by  Bloom,  continue  in  earnest.  As  cases  of  PL’s 
successful  implementation  begin  to  accrue,  educators  and  policy  makers  have  recognized 
that  a  paradigm  shift  may  be  necessary  for  our  education  system  to  keep  pace  with 
changing  times. 


21 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


22 


III.  TECHNOLOGY  TO  ENABLE  PERSONALIZED  LEARNING 


One  of  the  core  propositions  of  this  paper  is  that  technology  can  be  leveraged  for 
PL  in  order  to  enhance  the  effectiveness  of  military  training  and  education.  The  science 
behind  how  we  think  and  learn  has  advanced  in  recent  years,  and  technology  is  the  key  to 
capitalizing  on  that  knowledge  in  ways  not  previously  possible.  However,  we  should  not 
assume  that  more  is  better,  nor  use  technology  for  technology’s  sake.  The  pedagogy  must 
be  considered  first,  followed  by  the  right  technology  as  a  tool  to  enable  the  desired 
educational  approach  (Oblinger  &  Oblinger,  2005).  The  U.S.  Department  of  Education 
emphasized  the  impact  of  technology  in  learning  in  the  2010  National  Educational 
Technology  Plan: 

Just  as  technology  is  at  the  core  of  virtually  every  aspect  of  our  daily  lives 
and  work,  we  must  leverage  it  to  provide  engaging  and  powerful  learning 
experiences,  content,  and  resources  and  assessments  that  measure  student 
achievement  in  more  complete,  authentic,  and  meaningful  ways.  Tech- 
based  learning  and  assessment  systems  will  be  pivotal  in  improving 
student  learning  and  generating  data  that  can  be  used  to  continuously 
improve  the  education  system  at  all  levels.  (U.S.  Department  of  Education, 

2010,  p.  v) 

When  properly  implemented,  these  tools  have  the  power  to  maximize  student 
potential,  capturing  the  benefits  of  one-to-one  tutoring  that  Bloom  (1984)  articulated. 
Two  other  factors  that  technology  can  enhance  are  flexibility  and  responsiveness. 
Flexibility  refers  to  place  and  time,  bringing  the  instruction  to  the  point  of  need  at 
precisely  the  time  that  it  is  needed.  The  Navy’s  Ready  Relevant  Learning  initiative  seeks 
to  accomplish  this  by  leveraging  technology  to  re-engineer  technical  training  course 
content,  delivery  schedule,  and  delivery  modes  in  order  to  better  match  training  with  the 
needs  of  Sailors  (Department  of  the  Navy,  2016a).  PL  technologies  that  are  responsive, 
that  can  adjust  real-time  to  the  learner’s  preferred  style  or  speed,  prior  knowledge,  and 
aptitudes,  can  ensure  that  all  students  reach  their  maximum  potential  with  no  time  wasted 
and  often  at  a  cost  savings  (Dede,  2013). 

This  emphasis  on  both  effectiveness  and  efficiency  is  particularly  resonant  within 
DOD,  which  must  train  and  educate  more  than  2  million  active  and  reserve  service 
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members  and  800,000  civilians  (per  defense.gov),  at  a  cost  of  more  than  $6.5  billion 
annually  (DOD,  2015).  Since  the  1950s,  the  department  has  invested  heavily  in  the 
research  and  development  of  education  and  training  technology,  and  is  largely 
responsible  for  innovations  such  as  computer-assisted  instruction  (CAI)  and  computer- 
based  simulation  (Fletcher,  2009).  DOD  has  continued  to  pursue  technological  solutions 
such  as  Interactive  Multimedia  Instruction  (IMI),  Advanced  Distributed  Learning  (ADL), 
and  Intelligent  Tutoring  Systems  (ITS),  in  order  to  maximize  both  the  effectiveness  and 
efficiency  of  its  education  and  training  enterprise  (DOD,  2001b;  Fletcher,  2010).  In  the 
following  section,  I  will  discuss  these  and  other  programs,  systems  and  models  that  seek 
to  do  just  that. 

A.  TYPES  OF  TECHNOLOGY-BASED  INSTRUCTION 

1.  DOD-Interactive  Multimedia  Instruction  (IMI) 

Interactive  Multimedia  Instruction  (EMI)  is  a  broad  term  used  to  describe  a  variety 
of  interactive,  electronically  delivered  software  tools  for  instruction,  as  well  as  for 
training  management  and  support.  IMI  technologies  include:  interactive  courseware 
(ICW),  electronic  maintenance  publications  (e-pubs)  and  other  job  aids,  learning 
management  systems  (LMS),  simulator  trainers,  and  web-based  products  for  Advanced 
Distributed  Learning  (ADL).  These  systems  can  be  used  to  flexibly  deliver  interactive, 
competency-based,  and  individualized  multimedia  instruction.  IMI  are  intended  to 
capitalize  on  the  multimedia  effect,  varying  delivery  modes  during  instruction  in  order  to 
accommodate  differing  preferences  and  predispositions  in  the  maximum  number  of 
students.  Research  shows  that  this  can  be  done  with  success,  having  shown  an  average 
effect  size  of  .50  standard  deviations  (Fletcher,  2010). 

According  to  DOD  (2001b),  IMI  should  be  considered  when  a  large  number  of 
students  are  dispersed  over  time  and  place,  or  when  students  vary  in  experience  or  skill 
level.  This  prescriptive  statement  describes  nearly  every  imaginable  setting  for  training 
and  education  within  the  Navy.  Additionally,  IMI  can  be  designed  for  use  in  multiple 
levels  of  instruction,  which  could  considerably  decrease  overall  life  cycle  costs  due  to 
economies  of  scale.  However,  an  IMI  system  with  more  comprehensive  content  would 
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certainly  be  more  costly  to  develop.  An  in-depth  cost-benefit  analysis  is  a  critical  step  in 
the  process  to  design  and  implement  a  technology  to  replace  legacy  systems  and  methods 
of  instruction. 

2.  Advanced  Distributed  Learning 

First  introduced  by  DOD  in  1997,  the  purpose  of  the  Advanced  Distributed 
Learning  (ADL)  initiative  was  to  leverage  computer  technology  to  develop  and  distribute 
personalized,  cost-effective,  always  available,  high  quality  training  and  education  to 
service  members  and  DOD  civilians  (Wisher  &  Fletcher,  2004).  The  need  for  ADL 
developed  from  the  understanding  that  interactive,  engaging,  and  learner-centric 
approaches  to  education  were  likely  far  more  effective  than  the  traditional  instructor-led 
model,  and  from  the  recognition  that  ICT  was  sufficiently  advanced  and  ubiquitous  to 
capitalize  on  this  new  knowledge. 

The  advanced  distributed  learning  strategy  requires  re-engineering  the 
learning  paradigm  from  a  “classroom-centric”  model  to  an  increasingly 
“learner-centric”  model,  and  re-engineering  the  learning  business  process 
from  a  “factory  model”  (involving  mainly  large  education  and  training 
institutions)  to  a  more  network-centric  “information- age  model”  which 
incorporates  anytime-anywhere  learning.  (DOD,  2000,  p.  9) 

Figure  2  depicts  a  vision  of  this  information- age  model,  showing  how  ADL  could  be 
implemented  to  achieve  unprecedented  access  to  high-quality  learning  content. 
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Figure  2.  Theoretical  Network  Architecture  for  ADL.  Source: 

Wisher  &  Fletcher  (2004). 


3.  Intelligent  Tutoring  Systems 

Computer-assisted  instruction  and  computer-based  training,  the  foundational 
technologies  for  ADL,  are  systems  that  deliver  and  manage  educational  content  through 
the  use  of  computers,  usually  in  a  multimedia  format.  Given  today’s  proliferation  of  ICT, 
this  definition  includes  essentially  any  form  of  technology-based  educational  tool.  The 
underlying  “intelligence”  of  these  systems,  however,  varies  widely  across  four  general 
categories  of  CAI:  drill  and  practice,  tutorial,  tutorial  simulation,  and  tutorial  dialogue. 
Though  all  of  these  types  of  CAI  can  be  individualized  to  pace,  content,  sequence  or 
style,  only  tutorial  dialogue  attempts  to  explicitly  integrate  the  attributes  or  qualities  of  a 
one-on-one  teacher  to  student  relationship  (Fletcher,  1992b,  2010).  These  computer- 
assisted  instruction  programs  are  commonly  referred  to  as  Intelligent  Tutoring  Systems 
(ITS)  and  typically  incorporate  elements  of  artificial  intelligence  (AI)  and  machine 
learning  to  facilitate  the  understanding  of  student  level  of  knowledge,  response  to 
individual  needs,  and  teaching  based  on  learning  style  or  preferences,  all  in  real  time. 


26 


Another  goal  of  ITS  is  to  enable  mixed  initiative  dialogue,  or  the  ability  of  the 
system  to  carry  on  a  natural,  free  form  conversation  with  the  student  (Woolf  et  al.,  2013). 
One  analysis  of  a  group  of  recent  ITS  showed  an  average  effect  size  of  1.05  standard 
deviations,  showing  potential  to  meet  the  two  sigma  benchmark  as  the  technology 
progresses  (Fletcher,  2010).  With  the  exception  of  education,  AI  has  proliferated  through 
most  industries,  enhancing  growth  and  productivity.  Although  Al-enabled  ITS  are 
currently  capable  of  tailoring  learning  to  individual  needs,  they  are  not  yet  widely  used. 

Although  ITS  have  been  in  development  since  the  early  1980s,  exponential 
advances  in  computing  power  have  enabled  recent  breakthroughs  in  AI,  a  primary 
component  of  modern  ITS  (McArthur,  Lewis,  &  Bishay,  2005).  Today’s  intelligent 
learning  systems  can  understand  a  student’s  thought  processes,  motivations,  emotions, 
and  even  metacognition,  essentially  creating  a  mentor  for  every  learner.  Machine  learning 
and  data  mining  techniques  also  support  this  understanding  of  students  and  how  they 
learn  (Woolf  et  al.,  2013). 

ITS  are  generally  comprised  of  five  basic  components,  as  shown  in  Figure  3:  a 
student  model,  pedagogical  module,  domain  knowledge  module,  expert  model,  and 
communication  module  or  interface.  The  student  model  creates  a  unique  profile  based  on 
information  gathered  from  the  individual  learner.  This  component  tracks  the  student’s 
progress  by  measuring  changes  in  learning,  which  allows  the  pedagogical  model  to  adapt 
instruction  to  the  student,  whether  in  timing,  content,  or  delivery  method.  The 
pedagogical  model  makes  decisions  on  how  to  teach,  according  to  data  collected  by  the 
student  model.  The  domain  knowledge  module  contains  and  manages  a  representation  of 
all  facts,  concepts,  and  mental  models  of  the  information  that  is  being  taught.  The  expert 
model  compares  the  learner’s  responses  to  the  “ideal”  student’s  correct  solution  and 
recommends  any  required  remediation.  Finally,  a  communication  module  controls  all 
interactions  with  the  student  through  a  user  interface.  This  component  includes  visual 
layout  and  presentation  of  material,  as  well  as  dialogue  between  the  student  and  the 
system,  whether  textual  or  voice.  Throughout  the  learning  process,  the  first  four  elements 
are  in  constant  communication  with  one  another.  These  interactions  happen  at  machine 
speed,  keeping  the  student  engaged  in  real-time  via  the  communication  module.  Without 
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timely,  continuous,  and  meaningful  feedback  to  the  user,  the  true  value  of  an  ITS  may  go 
unrealized  (McArthur  et  al.,  2005;  Woolf  et  al.,  2013). 


Figure  3.  Components  of  an  Intelligent  Tutoring  System  (ITS).  Source: 

Beck,  Stern,  &  Haugsjaa  (1996). 

Because  of  their  inherent  complexity  and  the  need  for  such  in-depth 
representation  of  the  knowledge  contained  within,  development  time  and  costs  are  major 
limitations  of  ITS.  Estimates  as  high  as  200  man-hours  of  construction  time  per  one  hour 
of  instruction  highlight  the  requirement  for  significant  upfront  investments  to  develop  an 
ITS  (Fletcher,  2010).  While  the  savings  in  delivery  costs  once  the  system  is  in  place  can 
quickly  offset  development  costs,  the  time  horizon  to  realize  enough  economies  of  scale 
to  warrant  purchase  of  the  ITS  must  be  determined  on  a  case  by  case  basis.  Clearly,  there 
is  a  need  for  innovations  in  software  development  to  mitigate  potentially  prohibitive  costs 
of  constructing  an  ITS.  Two  possible  solutions  to  the  problem  are  common  authoring 
tools,  and  modularity.  Authoring  tools  would  allow  a  standardized,  straightforward 
development  platform,  which  could  allow  a  wider  range  of  users  the  ability  to  create  and 
edit  their  own  courseware.  A  system  of  modular  ITS  components,  created  using  a 
common  protocol,  would  allow  reuse  between  systems  and  developers,  greatly  reducing 
time  to  author  each  new  application.  These  “off-the-shelf’  modules  could  be  fit  together 
as  needed  to  create  a  combination  unique  to  each  specific  use,  without  the  need  to  build 
each  section  from  the  ground  up  (Beck  et  al.,  1996). 
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4.  Sharable  Content  Object  Reference  Model 

In  recognition  of  these  limitations,  the  ADL  initiative’s  goal  is  to  create  learning 
systems  that  are  interoperable,  reusable,  and  affordable.  As  such,  they  have  directed  the 
creation  of  a  common  architecture  and  set  of  standards  for  use  in  building  ITS  and 
supporting  elements.  The  Sharable  Content  Object  Reference  Model  (SCORM)  was 
developed  to  enable  learning  material  and  ITS  components  to  be  shared  across  various 
instructional  delivery  settings.  SCORM  is  an  important  first  step  in  the  adoption  of  an 
industry-common  format  for  ITS,  and  the  ADL  program’s  continuing  research  efforts  and 
collaboration  with  government  agencies,  industry,  and  academia  seek  to  make  that  a 
reality  (DOD,  2001a;  Wisher  &  Fletcher,  2004). 

5.  Learning  Management  Systems 

A  learning  management  system  (LMS),  the  backbone  of  any  distributed  learning 
system,  is  a  server-based  middleware  that  controls  delivery  of  content,  gathers  data,  and 
tracks  students’  progress  within  the  course  of  study  (Wisher  &  Fletcher,  2004).  The  LMS 
is  a  critical  component  of  ADL’s  vision  of  future  education  and  serves  to  connect  the 
learner  to  instructional  programs  and  content,  anytime  and  anywhere.  An  LMS  also 
allows  oversight  by  administrators  and  developers  to  keep  course  content  current  and  to 
track  and  repair  malfunctions  in  the  software.  The  LMS  communicates  directly  with 
course  management  system  (CMS),  which  serves  as  the  program’s  interface  with 
students.  The  CMS  launches  the  course,  tracks  progress,  and  provides  feedback  to  the 
student.  The  CMS  often  is  fully  integrated  within  an  ITS,  and  multiple  ITS  may  report  to 
a  single  LMS  for  overall  management  of  a  unit’s  training  and  education  program 
(DOD,  2001a;  Wisher  &  Fletcher,  2004). 

6.  Digital  Teaching  Platform 

A  Digital  Teaching  Platform  (DTP)  is  an  approach  to  technology  infrastructure 

that  incorporates  interactive  teaching  and  personalized  learning  technologies  into  a 

traditional  classroom  setting.  DTP  is  based  around  a  networked  digital  environment, 

where  every  student  and  the  teacher  have  computing  devices.  However,  whereas  ITS 

seem  to  imply  an  individually  guided  effort,  the  DTP  emphasizes  the  teacher’s  key  role 
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as  facilitator  and  administrator  of  learning.  Many  software  solutions  aim  to  make  the 
teacher  obsolete,  but  DTP  supports  current  infrastructure  and  classroom  paradigms, 
perhaps  easing  the  transition  to  an  all-digitally  moderated  learning  environment  in  the 
future  (Dede  &  Richards,  2013).  The  classroom  work  dynamic  is  flexible  with  DTP, 
allowing  for  both  individual  and  group  work,  whether  synchronous  or  asynchronous.  This 
ensures  the  ability  to  reinforce  a  social  constructivist  theory  of  learning,  and  aligns  with 
the  common  assertion  that  millennials  tend  to  prefer  working  in  groups  to  isolation  and 
individual  effort  (Dede,  2005;  Oblinger  &  Oblinger,  2005). 

Differing  from  an  LMS  where  the  software  manages  a  majority  of  functions,  DTP 
keeps  the  teacher  at  the  center  of  the  evolution,  responsible  for  curriculum  planning, 
student  learning  progress  assessment  and  remediation  management  (Dede,  2011).  The 
teacher’s  ability  to  monitor  engagement  and  interest,  to  foster  relationships  with  students, 
and  to  prescribe  interventions  as  needed,  along  with  the  technology’s  ability  to 
personalize  instruction  in  real  time,  give  timely  and  meaningful  feedback,  and  enable 
competency-based  progression  through  adaptive  computer-proctored  assessments  are  a 
combination  which  can  have  synergistic  effects  in  the  classroom.  Students  are  able  to  use 
technology  to  which  they  are  accustomed  in  their  daily  lives,  collaborate  and  interact  with 
peers,  and  still  receive  individual  support  from  the  teacher  in  an  as-needed,  focused 
manner  (Dede  &  Richards,  2013).  DTP  is  a  hybrid  approach  to  changing  the  paradigm  of 
learning  for  the  21st  century.  It  aims  to  capitalize  on  advances  in  ICT  and  PL  to  improve 
learning  outcomes,  even  in  the  face  of  budget  cuts  and  overcrowding,  while  retaining  the 
inherent  social  and  structural  benefits  of  instructor-led  classroom  environments. 

7.  Synthetic  Learning  Environments 

A  synthetic  learning  environment  (SLE)  replaces  or  augments  a  real-world 
interaction  within  the  context  of  a  computer-based  game,  simulation,  or  virtual-world 
experience.  The  ideal  SLE  applies  pedagogical  principles  along  with  accounting  for 
individual  learner  differences  to  deliver  an  optimal  experience  for  learning  (Cannon- 
Bowers  &  Bowers,  2008).  Games  and  simulators  have  long  been  used  as  training  devices, 
with  varying  degrees  of  sophistication  and  emphasis  on  sound  pedagogical  approaches. 
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Incorporation  of  these  legacy  tools,  along  with  budding  virtual  world  technologies  such 
as  virtual  and  augmented  reality,  are  allowing  SLE  to  become  legitimate  alternatives  to 
traditional  educational  models.  Vogel  et  al.  (2006),  in  a  meta-analysis  of  educational 
interactive  simulation  and  computer  game  research,  found  that  these  tools  did 
significantly  improve  learning  outcomes  over  conventional  teaching  methods. 

Several  factors  seem  to  impact  effectiveness  of  SLE,  and  among  these  are  level  of 
immersion  and  the  theory  of  situated  learning.  Immersion,  or  presence,  is  the  sense  or 
level  of  belief  that  one  actually  inhabits  a  simulated  event  or  place.  Higher  levels  of 
immersion  are  linked  to  increased  engagement  and  information  retention;  however,  the 
user’s  spatial  ability  may  be  an  important  moderator  in  this  effect.  Although  most  SLE 
can  be  easily  tailored  to  the  specific  needs  of  the  learner,  those  that  are  not  visually 
oriented,  or  who  are  not  as  comfortable  with  immersion  may  not  be  inclined  to  use  SLE, 
or  may  do  so  with  poor  results  (Cannon-Bowers  &  Bowers,  2008).  Virtual  and 
Augmented  reality  are  forms  of  SLE  that  provide  the  highest  levels  of  immersion. 

Situated  learning  theory,  with  roots  in  constructivism,  is  based  on  active 
participation  in  rich  contextual  scenarios,  guided  and  assessed  by  an  expert  mentor 
(Pritchard,  2014).  Learning  occurs  by  completing  authentic  learning  tasks,  so  named 
because  they  take  place  in  a  realistic  environment,  under  realistic  conditions,  and  with 
active  guidance  from  the  mentor.  Learners  are  more  likely  to  engage  at  a  deeper  level 
through  authentic  tasks  within  the  appropriate  context,  than  inauthentic  tasks  in  the 
wrong  context  (or  none  at  all),  such  as  a  traditional  classroom.  This  approach  to  learning 
in  essence  takes  Bloom’s  one-on-one  tutorial  relationship  to  the  next  level  by  placing  the 
instruction  in  context,  in  a  situated  environment.  This  concept  has  familiar  underpinnings 
in  the  world  of  simulation,  where  fidelity  of  the  experience,  both  physical  and 
psychological,  are  critical  to  the  level  of  transfer  achieved  by  the  trainee  (Cannon-Bowers 
&  Bowers,  2008).  Transfer,  or  the  ability  to  apply  in  the  real  world  what  is  learned  in  a 
training  environment,  is  maximized  using  experience-based,  engaging,  and  context- 
correct  instructional  methods  such  as  situated  learning  (Dede,  2005). 

Virtual  humans,  able  to  replicate  the  speech  patterns,  mannerisms,  and 

affectations  of  humans,  are  an  advanced  interface  used  for  creating  social  interactions  and 
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improving  presence  within  a  SLE.  Virtual  humans  use  advanced  modeling  and  graphical 
representation  of  human  movements,  artificial  intelligence,  speech  recognition,  and 
natural  language  processing  to  create  very  believable  pedagogical  agents.  These 
intelligent  agents  can  enhance  student  engagement  with  learning  material  by  creating 
rapport,  inciting  enthusiasm  or  motivation,  or  encouraging  a  struggling  student.  In  fact, 
people  tend  to  interact  with  virtual  humans  much  the  same  as  in  human-human 
engagements  (Swartout  et  al.,  2013).  While  accommodating  the  desire  for  social  contact 
by  learners,  virtual  humans  can  add  to  the  contextual  realism  and  effectiveness  of  SLE. 
Al-enhanced  virtual  humans  have  the  potential  to  become  the  mentor  for  every  student, 
closing  or  even  surpassing  the  two-sigma  gap,  and  are  an  exciting  development  to 
monitor  as  research  continues. 

While  highly  immersive,  SLE  inhabited  by  virtual  humans  is  another  tool  and  not 
a  panacea;  they  have  many  interesting  and  potentially  impactful  benefits  for  training  and 
education.  These  context-rich,  highly  adaptable  platforms  make  them  a  natural  fit  for  PL. 
Although  an  SLE  breaks  the  mold  of  traditional  school  settings,  perhaps  eventually 
rendering  it  all  obsolete,  it  does  not  imply  that  the  teacher  goes  away,  or  that  there’s  no 
social  interaction  between  peers.  In  a  virtual  world,  these  ideals  would  simply  take  on  a 
very  different  meaning. 

B.  CHALLENGES  AND  OPPORTUNITIES 

Many  of  these  technological  solutions  are  ripe  for  widespread  adoption,  but  there 
are  challenges  to  implementing  them  into  the  current  training  and  education  enterprise. 
One  important  implication  is  the  changing  role  of  the  teacher  that  would  occur  when 
shifting  to  a  learner-centered  paradigm.  PL,  by  nature,  is  typically  self-regulated  and 
computer  moderated.  Once  considered  a  ‘“sage  on  the  sage,”  or  primarily  a  transmitter  of 
knowledge,  the  instructor  would  become  “guide  on  the  side,”  as  more  of  a  facilitator  or 
mentor  for  PL,  and  allowing  the  technology  to  take  on  many  of  the  mundane  or  time¬ 
intensive  course  management  tasks,  such  as  creating  and  individualizing  learning  plans  to 
students’  needs  and  administering  regular  diagnostic  assessments  of  their  progress  (Dede, 
2013).  In  a  new  role  as  mentor  or  coach,  instructors  could  better  utilize  their  expertise  in 
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the  subject  matter  to  provide  context  for  students  and  foster  dialogue,  placing  less 
emphasis  on  teaching  skills  and  background  in  education.  This  point  is  particularly 
salient  for  the  military  environment,  where  service  members  are  given  instructional  duties 
based  on  operational  or  technical  experience,  not  on  teaching  credentials,  and  training 
time  for  instructional  techniques  is  limited  (Manacapilli  et  al.,  2011). 

Another  challenge  is  the  risk  of  disruption  of  current  established  educational 
norms  and  economics  in  schools  (Dede,  2013).  Some  techniques,  such  as  DTP,  promise 
to  seamlessly  integrate  into  today’s  classrooms,  but  nearly  all  technology-based  learning 
approaches  call  into  question  the  need  for  schools’  continued  existence  in  their  current 
form.  Doing  away  with  the  traditional  classroom  and  standard  school  day  raises  cultural 
and  political  barriers  to  acceptance.  Although  we  now  have  all  the  means  necessary  to 
implement  effective  PL  models  on  a  large  scale,  massive  reforms  and  reorganizations 
would  be  required.  The  implications  of  upending  the  traditional  paradigms  for  learning 
are  significant  and  not  fully  known.  For  these  reasons,  organizations  should  adopt  an 
incremental  approach  to  incorporating  technological  solutions  into  training  and  education 
programs,  and  only  those  solutions  that  are  built  upon  a  sound  pedagogical  foundation 
with  proven  results. 

C.  CHAPTER  SUMMARY 

Taking  all  of  these  challenges  into  consideration,  we  can  draw  some  useful 
conclusions  about  what  is  required  for  successful  implementation  of  technology-enabled 
PL  programs.  First,  the  technology  is  simply  a  resource  that  allows  the  learner-centered, 
tutorial  instruction  to  become  affordable  and  viable  (Grant  &  Basye,  2014).  It  erases  the 
economic  impossibility.  Next,  the  tech  enables  continuous  formative  assessments 
throughout  the  learning  process,  so  that  teachers  (or  the  program  itself)  may  address 
student  limitations  and  capitalize  on  strengths  in  a  timely  fashion  (Grant  &  Basye,  2014). 
That  being  said,  the  teacher’s  role  must  adjust  from  transmitter  of  knowledge  to 
facilitator  of  learning,  following  a  constructivist  theory. 

The  learner’s  progression  through  a  PL  program  should  be  competency-based, 
with  self-paced  advancement  occurring  only  after  demonstrating  mastery  of  the  material. 
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This  progression  is  achieved  by  providing  instruction  in  a  manner  best  suited  to  both  the 
subject  matter  and  to  the  individual  preferences  of  the  learner.  Finally,  the  learning 
environment  should  be  malleable,  facilitating  important  social  interactions  between  peers 
and  with  the  mentor,  while  enabling  individual  progress  through  the  content  (Grant  & 
Basye,  2014).  In  a  fully  integrated,  yet  supporting  role,  the  technological  solutions 
outlined  in  this  chapter  stand  ready  to  offer  new  and  more  efficient  ways  for  students  to 
receive  and  interact  with  educational  material,  optimizing  every  student’s  performance 
through  personalized  learning. 
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IV.  METHODOLOGY:  COST  ANALYSIS 


In  this  chapter,  I  present  the  different  types  of  cost  studies  that  may  be  used  to 
analyze  a  PL  program,  and  illustrate  how  cost-effectiveness  analysis  is  particularly  suited 
to  this  type  of  problem.  I  will  also  introduce  a  theoretical  model  of  the  costs  of  human 
tutoring,  lecture-based  classroom  instruction,  and  technology-based  PL  approaches  to 
learning,  with  the  intent  of  highlighting  the  differences  in  learning  approaches  in  terms  of 
fixed,  variable,  and  marginal  costs.  Finally,  I  will  discuss  methods  of  determining  the 
costs  and  benefits  of  educational  programs  for  comparison  in  a  cost-effectiveness 
analysis. 

A.  COST  STUDIES 

Cost  studies  combine  an  objective  and  quantitative  determination  of  program 
effects  with  a  thorough  and  detailed  evaluation  of  costs,  based  on  sound  economic 
principles.  Benefits,  or  effects  on  outcome,  are  compared  with  costs  through  a  variety  of 
methods  to  arrive  at  a  net  benefit  or  some  other  measure  of  value.  These  analyses  may  be 
used  for  a  variety  of  purposes,  such  as  comparison  of  multiple  courses  of  action, 
identifying  areas  for  improvement,  expansion  of  academic  understanding,  or  for  ensuring 
adherence  to  policy  requirements  (Ross,  Barkaoui,  &  Scott,  2007).  Although  there  are 
many  frameworks  and  methodologies  used  for  conducting  cost  studies,  they  all  have  a 
common  general  flow:  determine  the  objectives  of  the  study  and  alternatives  (if  any)  to 
be  compared,  decide  on  the  method  of  analysis,  measure  or  calculate  all  applicable  costs 
and  benefits,  compare  the  costs  and  benefits  to  find  the  net  benefit,  perform  a  sensitivity 
analysis  on  the  results,  and  recommend  a  course  of  action.  The  most  commonly  used 
types  of  cost  study  for  educational  evaluation  are  cost-benefit  analysis  (CBA)  and  cost- 
effectiveness  analysis.  The  primary  distinction  between  the  two  is  that  a  CBA  evaluates 
costs  and  benefits  converted  into  common  units,  usually  monetary,  while  a  cost- 
effectiveness  analysis  typically  compares  non-monetary  benefits  to  monetary  costs 
(Ross  et  al.,  2007). 
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When  measuring  the  costs  and  benefits  of  a  multi-year  program,  the  time  value  of 
money  must  be  considered.  This  process  recognizes  there  may  be  inflation  and  applies  a 
discount  rate  to  future  costs  and  benefits.  The  net  present  value  (NPV)  is  the  parameter  of 
interest,  as  it  expresses  all  current  and  future  costs  and  benefits  in  present  day  dollars  so 
that  programs  can  be  compared  side  by  side.  Additionally,  a  sensitivity  analysis  should  be 
completed  on  all  parameters  to  determine  robustness  of  findings.  This  is  easily  done  with 
a  model,  as  any  change  to  one  element  will  propagate  throughout.  Sensitivity  analysis 
also  helps  to  highlight  the  primary  cost  drivers,  which  further  aids  in  decision  making. 
Finally,  while  no  approach  to  cost  study  is  perfect,  any  known  limitations  of  the  model 
and  assumptions  made  must  be  explicitly  stated  at  the  outset. 

B.  TYPES  OF  COST  STUDIES 

In  order  to  make  a  direct  comparison  between  costs  and  benefits,  a  CBA  first 
converts  the  two  into  monetary  units.  From  here,  a  few  levels  of  analysis  are  possible. 
First,  the  benefit/cost  ratio  can  be  calculated  by  dividing  the  overall  value  of  the  result  by 
the  overall  cost.  Alternatively,  return  on  investment  (ROI)  may  be  expressed  as  a  ratio  of 
net  benefits  (total  value  minus  total  costs)  to  costs.  Although  a  result  of  greater  than  one 
indicates  positive  outcome  using  either  metric,  ROI  may  be  preferred  because  it  more 
easily  aligns  with  other  financial  reporting  criteria.  However,  there  may  be  variations 
using  either  method  depending  on  the  cost  model  used,  assumptions  made  in  converting 
units,  and  time  value  of  money  rates  (Fletcher  &  Sottilare,  2014).  Therefore,  a  cost- 
effectiveness  analysis  may  provide  a  clearer  understanding  of  the  true  cost-benefit 
relationship  of  an  educational  program,  whose  effectiveness  is  often  difficult  to  monetize 
(Fletcher,  1992a). 

One  important  distinction  to  make  in  a  cost-effectiveness  analysis  is  whether  the 
desired  outcome  is  to  reduce  time  to  train  (cost),  while  keeping  learning  objectives 
constant  (effectiveness),  or  to  improve  learning  outcomes,  keeping  time  constant.  With 
continuous  pressure  on  maintaining  both  fiscal  accountability  and  operational  readiness, 
military  and  government  organizations  nearly  always  seek  the  former  outcome  (Fletcher 
&  Sottilare,  2014).  Evaluating  cost-effectiveness  for  multiple  educational  systems  is 
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accomplished  by:  comparing  a  ratio  of  total  cost  to  effect  size,  minimizing  cost  to  show 
any  improvement  in  effect  size,  or  maximizing  effect  size  without  regard  to  cost.  No 
standard  decision  rule  criteria  exists  for  technology-enabled  PL  systems,  but  in  general,  a 
program  is  worth  adopting  if  it  demonstrates:  greater  effectiveness  for  the  same  cost  than 
the  alternatives  or  status  quo,  the  same  effectiveness  at  lower  cost,  or  the  highest  cost- 
effectiveness  ratio  (Ross  et  al.,  2007).  The  decision  space  for  comparing  a  single  program 
to  the  status  quo  is  depicted  in  Table  1,  with  question  marks  indicating  an  ambiguous 
result  and  need  for  further  analysis.  Of  note,  the  sequence  of  measuring  effectiveness  and 
costs  of  a  program  is  important,  as  a  decrease  in  effectiveness  from  the  status  quo 
requires  rejection  without  need  to  spend  time  and  resources  calculating  the  program’s  full 
costs  (Fletcher  &  Sottilare,  2014). 


Table  1.  Decision  Space  for  Cost-Effectiveness  Assessment.  Source: 

Fletcher  &  Sottilare  (2014). 


Effectiveness 


Costs 


Increase 

Decrease 

Increase 

?? 

Accept 

Decrease 

Reject 

?? 

C.  DETERMINING  COST  OF  INSTRUCTIONAL  PROGRAMS 

Overall  cost  calculations  for  instructional  programs  can  fluctuate  greatly  based  on 
what  assumptions  are  made  and  the  processes  used  to  ascertain  and  measure  all  cost 
components  (Fletcher,  1992a).  These  assumptions  and  procedures  must  be  explicit  in  any 
evaluation.  To  organize  and  define  all  cost  elements,  Fletcher  (1992a)  recommends 
starting  with  an  accepted  framework  of  cost  components,  such  as  the  Knapp  and 
Orlansky  model  (1983).  This  model  breaks  down  the  main  elements  of  life-cycle  costs 
(FCC),  R&D,  initial  investment,  operations  and  maintenance,  and  disposal,  further  into 
their  hypothetical  supporting  elements.  Although  the  model  contains  75  separate  cost 
components,  not  all  are  relevant  or  applicable  to  instructional  systems.  For  instance, 
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R&D  and  initial  investment  costs  are  likely  sunk  for  an  existing  program  and  wouldn’t  be 
included.  Likewise,  disposal  and  salvage  costs  are  typically  minor  for  instructional 
systems  and  often  omitted  from  analysis  (Fletcher  &  Sottilare,  2014). 

Fletcher  and  Sottilare  (2014)  developed  another  useful  model  for  classifying 
training  and  education  costs  based  on  earlier  work  by  Levin  (1983)  and  Kearsley  (1982). 
The  framework  uses  six  cost  categories:  personnel,  facilities,  equipment,  consumables, 
other  direct  costs,  and  indirect  costs.  Personnel  costs  include  any  human  resources 
required  to  administer  the  program,  while  facilities  costs  refer  to  the  physical  space  used. 
Equipment  includes  the  instructional  materials,  software,  and  other  physical  furnishings, 
and  consumables  are  items  that  are  used  up,  such  as  office  supplies.  Other  direct  costs, 
including  client  inputs,  would  include  opportunity  costs  incurred  by  the  student  or 
employer.  For  example,  in  a  military  setting,  this  category  would  include  the  student’s 
pay  and  benefits  paid  by  the  sponsor  organization  during  training,  also  a  primary  cost 
driver  (Angier  &  Fletcher,  1992).  Indirect  costs  covers  shared  operating  costs, 
administrative  expenses,  and  overhead,  and  these  costs  may  be  characterized  as  fixed, 
variable,  or  mixed.  The  six  cost  categories  can  be  further  organized  across  the  temporal 
phases  of  instructional  systems  development:  analysis,  design,  development, 
implementation,  and  evaluation,  creating  30  separate  “bins”  (Fletcher  &  Sottilare,  2014). 

Using  one  of  above  two  frameworks  for  cost  analysis,  the  educational  evaluator  or 
economist  can  readily  identify,  classify,  and  then  analyze  all  parts  of  the  cost  side  in  a 
logical,  coherent  manner.  While  this  analysis  may  be  thorough,  and  may  accurately 
compare  a  PL  program  with  the  status  quo,  it  gives  us  little  indication  as  to  the 
relationships  and  differences  between  cost  components  within  each  model  as  a  whole. 
Understanding  how  the  cost  curves  behave  can  give  the  decision  maker  a  clearer  idea  of 
the  potential  long-term  benefits  of  a  technology-based  PL  approach. 

D.  THEORETICAL  MODEL 

Analyzing  the  relationship  between  cost  components,  such  as  fixed,  variable, 
average  total,  and  marginal  costs  provides  another  perspective  for  understanding  the 
differences  between  human-tutoring,  teacher-led  classrooms,  and  technology-enabled 
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personalized  learning.  Economies  and  diseconomies  of  scale  become  more  apparent,  as 
do  the  differences  in  unit  costs  of  education.  Also,  we  can  better  illustrate  the  benefits 
provided  by  PL,  by  describing  each  student’s  effective  learning  level  as  a  marginal 
benefit.  This  basic  microeconomic  assessment  of  cost  curves  is  intended  to  serve  as  a 
theoretical  model  for  comparing  and  contrasting  the  cost  behavior  of  differing  approaches 
to  instruction. 

Beginning  with  the  oldest  and  most  straightforward  approach,  the  Aristotle  model 
assumes  that  a  single  teacher  is  assigned  to  every  student.  There  are  very  low  fixed  costs, 
as  this  one-to-one  tutoring  doesn’t  necessarily  require  a  classroom  or  high-tech 
instructional  materials.  The  major  cost  is  the  teacher’s  compensation,  which  is  variable, 
so  that  with  every  additional  student  trained,  the  organization  must  pay  another  full 
salary.  The  average  total  cost  curve  is  flat,  and  relatively  high,  showing  no  economies  of 
scale.  However,  the  marginal  benefit  remains  relatively  constant,  as  every  student  gets  a 
teacher  full-time.  As  these  are  human  teachers  with  differing  levels  of  skill  and 
knowledge,  this  curve  the  marginal  benefit  is  likely  different  for  each  student.  As  long  as 
the  marginal  benefit  exceeded  the  marginal  cost,  and  there  were  no  constraints  on  the 
budget,  this  approach  would  make  sense  to  implement. 

The  industrial  model,  or  teacher-led  classroom  approach,  is  next.  Fixed  costs  are 
generally  comprised  of  school  buildings,  furniture  and  instructional  equipment.  This 
fixed  cost  curve  is  stepped;  when  enough  students  are  added  to  require  expansion,  you 
need  a  bigger  school,  more  classrooms,  more  equipment,  etc.  The  curve  is  flat  again  until 
the  school  reaches  capacity.  Curriculum  has  already  been  developed  and  is  a  sunk  cost. 
Variable  costs  of  students  can  be  significant,  including  cost  of  travel  to  the  school,  extra 
TAD  pay,  lodging,  and  materials  and  other  expenses.  Variable  costs  of  instructors  will 
behave  in  a  stepped  manner,  as  more  teachers  are  needed  when  enough  students  are 
added  to  require  an  additional  classroom. 

While  there  are  small  economies  of  scale  initially  and  after  each 

classroom/teacher  addition,  we  likely  see  constant  return  to  scale.  However,  one  could 

imagine  that  diseconomies  of  scale  to  arise  when  the  school  is  so  large  and  overcrowded 

that  it  struggles  to  operate  efficiently,  requiring  additional  infrastructure  and 

39 


administrative  investments.  Variable  costs  are  a  primary  driver  of  total  costs,  which 
increase  rapidly  with  student  population.  Benefits  start  at  the  same  level  as  the  previous 
one-to-one  model,  but  as  students  are  added  to  a  classroom,  this  benefit  begins  to 
decrease  until  the  classroom  reaches  capacity  and  another  classroom  is  added  and  another 
teacher  hired.  The  marginal  benefit  will  saw  tooth  with  student  to  teacher  ratio,  but  would 
likely  not  return  to  the  beginning  state  of  1:1,  reflecting  the  assumption  that  higher 
student-teacher  ratios  negatively  affect  learning  outcomes. 

Although  PL  can  be,  and  has  been  applied  directly  to  existing  physical 
classrooms,  its  potential  is  better  highlighted  when  applied  via  the  ADL  concept,  where 
instruction  is  distributed  to  the  point  of  need  via  virtual  classrooms.  Therefore,  one 
notable  absence  from  variable  costs  of  the  PL  model  would  be  travel  and  TAD  expenses. 
While  human  mentors  are  still  a  part  of  PL,  higher  ratios  would  be  acceptable,  as  the 
technology  manages  much  of  the  administrative  burden.  Due  to  the  network  effects  of 
economies  of  scale  and  distribution  of  instructional  programs  away  from  traditional 
schoolhouses,  the  high  upfront  development  costs  of  an  ITS  are  quickly  diluted  over 
increasing  numbers  of  students,  while  distribution  costs  remain  relatively  stable,  with 
only  slight  increases  when  extra  servers,  human  mentors,  or  system  administration  cost 
steps  are  reached.  The  combined  effect  of  network  economies  of  scale  and  low  delivery 
cost  through  distribution  of  learning  lead  to  a  very  low  marginal  cost  per  student,  and  a 
long-term  average  total  cost  much  lower  than  the  industrial  model. 

The  PL  model  captures  all  of  the  benefits  of  Aristotle  model,  but  illustrates  that  a 
significant  economy  of  scale  exists  when  variable  delivery  costs  are  kept  low.  Although 
initial  investment  into  PL  is  significant,  often  requiring  as  much  as  200  man-hours  and 
$50,000  to  develop  one  hour  of  instruction,  these  economies  of  scale  are  best  realized 
when  PL  is  used  to  transform  courses  with  longest  lengths  and  highest  throughput  of 
students  (Fletcher,  2010).  All  three  models  point  out  the  critical  concept  that  long-run 
average  total  costs  are  most  sensitive  to  delivery  of  instruction,  not  content  development 
or  school  infrastructure.  Fewer  human  teachers,  less  travel  time  and  cost,  more  cost- 
effective  delivery  of  training,  and  a  consistently  high  level  of  student  performance  make 
PL  a  seemingly  attractive  alternative  to  the  industrial  model,  although  candidate 
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programs  should  be  analyzed  on  a  case-by-case  basis  to  determine  individual  viability 
before  conversion. 

E.  OPPORTUNITY  COSTS 

Opportunity  cost  refers  those  costs  incurred  or  things  that  must  be  given  up  to 
acquire  something,  and  they  may  either  be  easily  measurable  or  more  qualitative  in 
nature.  In  terms  of  comparing  instructional  models,  I  will  discuss  what  is  given  up  by 
choosing  the  traditional  lock-step  classroom  approach  over  technology-based  PL,  and 
vice  versa.  While  not  always  readily  apparent,  study  of  these  opportunity  costs  is  a  vital 
component  of  any  comprehensive  comparison  of  alternative  programs. 

Time  savings  is  perhaps  the  most  rigorously  proven  and  highest  payoff  benefit  for 
technology-based  self-paced  PL.  With  PL,  faster  completion  rates  result  from  pace, 
content,  and  sequence  of  instruction  all  being  tailored  to  individual  student  needs. 
Compare  this  to  the  lock-step  approach,  which  keeps  groups  of  students  together  in  time, 
location,  and  learning  rate  as  a  class.  Carey  et  al.  (2007)  found  that  converting  Navy 
initial  technical  training  (A-school)  courses  to  computerized  self-paced  format  reduced 
time  to  train  by  10-31  percent,  and  that  performance  in  follow-on  training  was  not 
adversely  affected.  Fletcher’s  (2010)  continuing  analysis  has  shown  a  conservative  rule- 
of  thumb  being  a  30  percent  savings  in  training  time  for  adopting  technology-based 
instruction.  A  primary  driver  for  cost  of  training  is  student  pay  and  allowances,  so 
reducing  time  to  train  has  always  been  a  goal  of  instructional  systems  developers.  In 
order  to  determine  the  opportunity  costs  incurred  due  to  longer  time  to  train  with 
traditional  instruction,  a  rate  for  the  daily  ‘cost  of  a  Sailor’,  called  the  Individuals 
Account  programming  rate  can  be  used.  To  calculate  this  cost,  multiply  the  rate  given  for 
the  current  year  (in  2007,  it  was  $148/day)  by  the  difference  in  days  between  the  lengths 
of  alternative  programs  (Carey  et  al.,  2007).  To  maximize  the  effective  savings  from  this 
opportunity  cost,  we  should  target  courses  with  longer  current  length  and  higher  student 
throughput  for  conversion  to  PL. 

As  PL  solutions  tend  to  be  more  portable  and  flexible  than  traditional  learning, 
other  potential  savings  occur  when  training  is  distributed  to  point  of  need,  saving  travel, 
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TAD,  and  lost  work  costs.  Though  less  straightforward  than  time-to-train  savings,  these 
opportunity  costs  are  also  quantitative  and  easily  calculated  (Fletcher,  2010).  Another, 
less  apparent  benefit  of  technology-based  is  the  inherent  ability  for  data  collection 
compared  to  traditional  models.  This  data  enables  a  better  understanding  of  the  optimal 
amount  of  training  needed  for  an  individual,  including  how  long  that  training  should  take 
(Angier  &  Fletcher,  1992).  These  insights  can  then  be  applied  to  further  refine  the 
courseware  development  process  as  PL  approaches  gain  ubiquity.  Technology-enabled, 
flexibly  distributed  PL  is  a  force  multiplier  that  enhances  operational  readiness,  while 
maintaining  or  reducing  personnel  costs.  Reducing  time-to-train  gets  people  back  to  work 
faster,  thereby  increasing  readiness  with  same  force  size  (Fletcher,  2010). 

Adoption  of  self-paced  learning  approaches  is  not  without  opportunity  costs.  The 
traditional  military  classroom  has  some  qualitative  benefits  that  may  be  largely  excluded 
from  PL  and  are  worthy  of  mention.  For  instance,  students’  exposure  to  experienced 
instructors  and  one  another  throughout  a  course  affords  additional  time  for  military 
training,  discipline,  and  camaraderie,  which  is  especially  critical  early  in  a  career 
(Fletcher,  2010).  Team-building  exercises,  “sea  stories,”  and  experience-based  contextual 
examples  are  natural  elements  of  today’s  military  classroom,  but  developers  of  PL  will 
need  to  find  new  ways  achieve  these  benefits.  “Sailorization”  is  a  process  by  which  the 
Navy  imbues  culture,  values,  and  traditions  into  its  members  and  ensures  a  foundation  for 
teamwork,  pride  and  professionalism.  This,  along  with  unit-cohesion  and  esprit  de  corps 
are  intangible  aspects  of  the  naval  service  that  are  vital  to  success,  and  any  PL  initiative 
would  need  to  be  careful  to  maintain  these  attributes  that  a  group  of  students  would 
normally  develop  during  training. 

Another  potential  opportunity  cost  to  implementing  PL  occurs  when  self-paced 
blocks  of  instruction  are  combined  with  lock-step  modules.  In  these  cases,  the  full 
benefits  of  PL  are  not  likely  to  be  realized,  as  faster  learners  must  wait  on  slower  learners 
in  order  to  continue  training.  This  creates  a  free  resource  problem,  and  the  potential  exists 
for  negative  outcomes  (Angier  &  Fletcher,  1992).  One  solution  would  have  early 
finishers  of  self-paced  modules  continue  learning  towards  an  even  higher  level  of 
understanding.  Alternatively,  they  may  be  allowed  to  undertake  additional  qualifications 
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or  be  incentivized  to  assist  in  the  progress  of  slower  learners.  This  issue  would  need  to  be 
evaluated  on  a  case-by-case  basis  early  in  the  design  process,  when  a  simple 
reorganization  of  course  structure  might  alleviate  the  problem  altogether. 

All  of  these  opportunity  costs  represent  tradeoffs  that  are  likely  to  occur  when 
deciding  between  alternative  instructional  approaches.  Course  designers  and  evaluators 
should  be  aware  of  these  elements  in  order  to  measure  and  compare  them,  and  also  to 
incorporate  new  methods  and  techniques  that  could  offset  the  costs.  Tradeoffs  are  a  part 
of  any  economic  analysis,  and  must  be  fully  acknowledged  if  the  best  decision  is  to  be 
made. 

F.  DETERMINING  THE  BENEFITS  OF  INSTRUCTIONAL  PROGRAMS 

The  primary  benefit  of  an  instructional  program  is  its  level  of  effectiveness  at 
achieving  learning  objectives.  In  this  section,  I  will  discuss  two  widely  accepted  methods 
for  evaluating  and  comparing  the  effectiveness  of  these  programs:  effect  size  and  the 
Kirkpatrick  Model.  Effect  size  is  most  commonly  used  to  compare  study  results  in  the 
realm  of  education,  while  the  Kirkpatrick  Model  has  been  primarily  implemented  in 
organizational  training  settings.  While  either  method  can  easily  adapt  to  either  education 
or  training  evaluation,  availability  of  data  and  the  type  activity  being  assessed  will  dictate 
which  is  chosen.  Effect  sizes  are  most  readily  derived  from  results  of  highly  controlled, 
academically  rigorous  studies  or  experiments.  The  Kirkpatrick  Model  is  designed  to 
capture  data  from  multiple  perspectives,  from  staged  experiments  to  real-world 
application  of  training.  That  said,  there  are  aspects  of  each  approach,  which  could  even 
be  mixed  together,  as  the  situation  allows,  reaching  a  more  comprehensive  assessment  of 
program  effectiveness. 

1.  Effect  Size 

Across  the  academic  literature,  effect  size  is  most  often  used  to  explain  the  results 

of  a  particular  intervention  as  compared  the  status  quo  or  to  alternative  interventions. 

This  method  is  given  so  much  leverage,  that  often,  the  simple  finding  of  a  statistically 

significant  effect  by  a  study  that  meets  standards  of  rigor  and  control  is  evidence  enough 

to  prove  a  program  worthwhile  (Ross  et  al.,  2007).  While  there  are  some  inherent 
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problems  with  this  singular  emphasis,  the  concept  is  well  rooted  in  the  literature.  Hattie’s 
(1999)  seminal  work  synthesized  effect  size  data  from  337  meta- analyses,  covering 
180,000  studies  and  200,000  effect  sizes  to  conclude  the  average  effect  of  educational 
interventions  and  innovations  to  be  .40  standard  deviations.  Along  with  Bloom’s  (1984) 
findings  that  one-to-one  tutoring  produced  an  outcome  of  2.0  standard  deviations  above 
the  mean,  these  benchmarks  have  helped  shape  the  interpretations  of  effect  size  into  the 
generally  accepted  standards  shown  in  Table  2.  Additionally,  each  range  of  effect  sizes  is 
explained  in  the  table  by  the  percentile  increase  for  a  theoretical  student  at  the  50th 
percentile  before  the  intervention.  For  example,  an  effect  size  of  .5  is  an  improvement  of 
a  student  from  50th  percentile  to  69th  percentile,  while  an  effect  of  2.0  raises  that  student 
to  the  98th  percentile  (Fletcher,  1992a). 

Table  2.  Interpreting  Effect  Size  of  Educational  Intervention  Results. 

Adapted  from  Fletcher  &  Morrison  (2014). 


Effect  Size 

Suggested  Interpretation 

50th  Percentile  Raised  To  (Est.) 

<0.25 

Negligible 

59th  percentile 

0.25-0.40 

Small 

60th-65th  percentile 

0.40-0.60 

Moderate 

66th-72nd  percentile 

0.60-0.80 

Large 

73rd-78th  percentile 

>0.80 

Very  Large 

79th  percentile  and  up 

>2.00 

1:1  Tutoring  Benchmark 

98th  percentile  and  up 

In  order  for  an  effect  size  to  have  any  true  meaning,  the  study  by  which  it  was 
found  must  be  of  a  quality  such  that  the  findings  can  be  trusted.  The  level  of  rigor  and 
attention  to  appropriate  methodology  must  be  present  in  the  research  before  evaluators 
rely  on  this  metric  for  a  cost-effectiveness  analysis. 
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2.  Kirkpatrick  Model 

Kirkpatrick’s  Four  Levels  Evaluation  Model  is  another  widely  accepted  method 
for  evaluating  the  effectiveness  of  training  for  a  variety  of  organizational  types 
(Kirkpatrick  &  Kirkpatrick,  2006).  The  levels  are:  1-Reaction,  2-Learning,  3-Behavior, 
and  4-Results,  and  are  intended  to  be  followed  sequentially,  beginning  with  reaction  and 
continuing  to  results,  as  resources  allow.  Each  successive  level  builds  upon  what  has 
been  learned  in  the  previous  steps.  Reaction  is  the  immediate  post  training  assessment  of 
student  satisfaction,  often  recorded  via  survey.  The  learning  level  assesses  the  degree  to 
which  training  objectives  were  achieved,  or  whether  the  student  acquired  the  knowledge 
or  skills  intended.  A  post-training  test  on  these  objectives  would  be  the  simplest  way  to 
accomplish  a  Level  2  assessment.  Level  3,  behavior,  refers  to  the  application  of  learned 
knowledge  or  skills  to  the  work  environment.  Knowing  from  Level  2  that  the  training 
was  successful,  Level  3  helps  to  determine  if  it  is  the  right  training  for  the  job  being 
performed,  and  how  well  the  student  has  transferred  training  to  performance.  Behavior  is 
more  difficult  to  measure,  and  is  often  assessed  through  surveys,  interviews,  or 
observations  of  the  worker  and  their  supervisor.  Linally,  the  results  level  seeks  to 
determine  whether  the  training  and  subsequent  transfer  of  acquired  knowledge  and  skill 
produces  any  tangible  benefit  to  the  organization.  Results  are  often  the  most  troublesome 
metrics  to  capture,  as  many  organizations  do  not  have  the  data  tracking  capability  to 
determine  causality  of  changes  in  productivity,  and  DOD  is  no  exception. 

G.  CHAPTER  SUMMARY 

Measuring  the  benefits  of  technology-enabled  PL  programs  is  critical  to  an 
accurate  cost-effectiveness  analysis,  and  can  be  accomplished  by  using  effect-size 
comparison  or  the  Kirkpatrick  Model,  two  tried  and  true  techniques.  While  both  methods 
are  backed  by  decades  of  research  and  peer-reviewed  publications,  the  onus  remains  on 
the  evaluator  to  ensure  that  the  data  collected  is  done  so  in  an  academically  rigorous 
manner.  Biases  and  external  influences  should  be  recognized  and  controlled  for  whenever 
possible.  In  the  end,  the  most  important  consideration  is  that  the  programs’  results  are 
being  compared  in  like  units  and  using  the  same  methodologies. 
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V.  DIGITAL  TUTOR  CASE  STUDY 


The  U.  S.  Navy’s  Digital  Tutor  (DT)  program  is  an  ITS  currently  in  use  in  the 
initial  technical  training  phase  for  Sailors  in  the  Information  Systems  Technician  (IT) 
rating.  The  program  is  in  the  midst  of  a  three-year  pilot  contract,  being  administered 
alongside  the  traditional  IT  training  path  in  Pensacola,  FL.  DT  represents  one  of  the  most 
advanced  PL  initiatives  being  undertaken  by  the  military,  and  the  program  showed 
promising  early  results.  However,  a  recent  comparison  of  DT-trained  with  traditionally 
trained  Sailors  in  the  Fleet  has  raised  some  questions  as  to  the  effectiveness  of  DT, 
threatening  the  program’s  continued  existence.  In  this  section,  I  will  familiarize  the 
reader  with  the  DT  program,  from  conception  to  implementation  and  testing.  I  will  then 
review  two  recent  studies  from  the  Institute  for  Defense  Analyses  (IDA)  and  NETC. 
Finally,  I  will  reconcile  the  findings  of  these  studies  with  cost  principles  and  theoretical 
models  mentioned  above  to  recommend  a  course  for  further  evaluation  of  the  program. 

Digital  Tutor  began  as  a  DARPA  project,  intended  to  accelerate  the  development 
of  expertise  from  a  novice  beginning  state  by  compressing  years  of  training  and 
experience  into  a  short  time  period  of  training  using  an  ITS.  In  other  words,  the  original 
goal  of  DT  was  to  produce  a  significantly  improved  training  outcome  in  a  fixed  time,  but 
shorter  than  the  traditional  training  path.  The  purpose  of  modern  ITS  and  specifically  DT 
is  to  operate  as  a  human  tutor  would,  except  to  do  so  cost  effectively  using  information 
structures  and  tutorial  dialogue  imbedded  within  a  computer  tutor.  By  capturing  the 
teachings  of  subject  matter  experts,  also  expert  instructors,  and  synthesizing  them  for 
computer  delivery,  the  system  prepares  students  for  problems  they  will  likely  encounter 
in  the  real  world,  in  addition  to  imparting  declarative  knowledge  (Fletcher  & 
Morrison,  2014). 

About  half  of  DT’s  development  costs  went  to  fund  the  human  tutor-created 
coursework,  which  was  the  basis  for  designing  its  software  and  enabling  economic 
scalability.  Once  the  human-tutored  course  was  created,  Content  Authors  worked  with  a 
Content  Engineer,  an  AI  expert  with  a  deep  understanding  of  DT  and  its  software 

architecture,  to  develop  the  instructional  flow  of  the  program.  Behind  the  scenes,  an 
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Inference  Engine  derives  the  problem-solving  process  being  used  by  each  individual 
learner.  These  findings  are  passed  to  the  Instruction  Engine,  which  determines  what 
elements  should  be  presented  next  and  when  to  conduct  an  assessment  of  the  learner’s 
knowledge.  The  Conversation  Module  is  the  interface  with  the  student,  engaging  with 
tutorial  dialogue  in  natural  language.  It  also  delivers  the  assessments  as  recommended  by 
the  previous  two  modules,  and  contacts  a  human  monitor/mentor  if  suggested  by  the 
Recommender.  All  of  the  training  is  set  in  the  context  of  actual  systems  used  in  the  Fleet, 
providing  a  situated  learning  component  to  the  process  (Fletcher  &  Morrison,  2014). 

DT  uses  varied  approaches  to  instruction,  depending  on  complexity  of  the 
problem.  “Frequent  and  substantive  dialogue  interaction  with  learners,  authentic,  situated 
problem-solving,  continual  diagnostic  assessment  of  individual  learning  and  progress, 
required  reflection  on  concepts  illustrated  by  problem  content  and  processes,  and 
integration  of  human  monitors  and  mentors”  are  components  that  enable  deep,  yet 
accelerated  learning  (Fletcher  &  Morrison,  2014,  p.  11).  Additionally,  along  with  a 
monitoring  function,  the  human  instructors  are  considered  instrumental  to  the  DT 
experience,  able  to  inject  ‘sea  stories’  or  personal  experiences  based  in  the  context  of  the 
Fleet.  This  includes  familiarization  time  for  culture,  traditions  and  practices  of  the  Navy. 

DT  differs  from  fully  self-paced  programs,  in  that  the  overall  time  of  instruction  is 
fixed.  In  DT,  if  a  student  finishes  a  module  with  extra  time,  they  are  given  increasingly 
difficult  problems,  thereby  achieving  a  higher  level  of  understanding  than  the  minimum. 
Even  so,  the  overall  pace  of  learning  could  be  increased  to  allow  all  levels  of  students  to 
progress  quicker  than  traditional  method.  As  incorporated  into  Navy  IT  A-school,  the  DT 
module  shortens  time-to-train  by  10  weeks.  While  the  program  was  developed  to  teach  a 
generic  IT  professional  training  course  in  18  weeks,  it  does  not  include  Navy-specific 
elements  of  IT  training.  Therefore,  the  Navy  version  includes  a  5-week  hybrid  course  at 
the  beginning  and  a  4-week  hybrid  course  at  the  end,  for  a  total  time  to  train  of  27  weeks. 
The  traditional  track  requires  37  weeks  to  produce  the  same  Navy  Enlisted  Classification 
(NEC)  with  a  6-year  obligation  (Department  of  the  Navy,  2016b).  The  A-school  course 
(19  weeks)  for  4-year  obligation  sailors,  a  separate  NEC,  was  not  studied. 
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A. 


INSTITUTE  FOR  DEFENSE  ANALYSES  REPORT 


At  four  points  throughout  the  development  of  DT,  the  Institute  for  Defense 
Analyses  (IDA)  conducted  formative  assessments  of  the  program’s  effectiveness,  with  a 
fifth,  summative  evaluation  occurring  at  the  end  using  the  entire  16- week  program.  The 
final  assessment  compared  DT  graduates  to  graduates  of  the  traditional  Information 
Technology  Training  Continuum  (ITTC)  and  to  Fleet  ITs,  with  an  average  of  9.6  years  of 
experience,  in  a  series  of  exercises  over  2.5  days  meant  to  replicate  complex  problems  of 
the  type  which  might  be  encountered  in  the  Fleet.  Overall,  the  DT  subjects  drastically 
outperformed  the  ITTC  and  Fleet  groups,  in  some  cases  by  a  substantial  margin,  all 
statistically  significant.  These  results  lead  the  authors  to  conclude  that  it  is  possible  to 
develop  real  expertise  in  a  drastically  accelerated  timeline  and  via  ITS  technical  training 
only  (Fletcher  &  Morrison,  2014). 

In  terms  of  assessing  DT,  the  IDA  evaluators  sought  to  determine  whether  the 
program  instilled  in  its  graduates  fleet- appropriate  IT  skills  and  knowledge,  and  whether 
they  were  superior  to  those  provided  by  classroom  instruction  and  to  the  skills  and 
knowledge  demonstrated  by  Fleet-experienced  ITs.  The  groups  being  evaluated  were 
notably  small,  with  only  12  participants  from  each  cohort.  Overall  results  from  the 
assessment  weighed  quite  convincingly  in  DT’s  favor.  DT  graduates  outperformed  ITTC 
and  Fleet  teams  on  troubleshooting  exercises,  solving  more  difficult  problems  at  a  higher 
success  rate.  Additionally,  DT-taught  students  fared  much  higher  on  knowledge  tests  than 
both  sets  of  counterparts.  In  all  cases,  the  effect  sizes  of  DT  over  the  status  quo  were 
noteworthy,  with  six  of  14  being  over  the  2.0,  “Bloom’s  challenge”  threshold.  The  DT 
group  showed  weakness  in  only  one  segment  of  the  test,  the  security  exercise,  being 
bested  by  the  Fleet  group,  although  not  as  badly  as  the  Fleet  group  outperformed  the 
ITTC  group.  This  shortfall  was  determined  to  stem  from  the  DT  program’s  content 
development,  where  an  acceptable  expert  in  network  security  was  not  available  for 
digitization  of  that  module.  Nonetheless,  the  results  do  suggest  an  overwhelming 
superiority  of  DT  to  ITTC  for  initial  training  of  ITs. 

On  the  other  side  of  this  result,  Fleet  ITs  were  likely  at  a  knowledge  disadvantage, 


as  all  follow  on,  refresher,  and  sustainment  training  is  known  to  vary  widely  across  the 
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Fleet  and  over  time.  Also,  an  IT’s  actual  daily  duties  are  often  specialized  within  their 
unit,  allowing  for  less  experience  across  the  array  of  potential  IT  jobs.  In  essence, 
students  fresh  out  of  training,  whether  DT  or  ITTC  would  be  more  competent  and  more 
comfortable  with  the  format  of  the  exercises  seen  in  this  study  as  well  as  with  the 
knowledge  tests.  For  these  reasons,  among  others,  the  comparison  between  DT  and  Fleet 
Sailors  is  not  adequate  to  conclude  that  the  program  creates  expertise  comparable  to  that 
of  a  Sailor  with  nearly  10  years  of  experience.  The  comparisons  between  the  DT  and 
ITTC  groups  should  carry  more  weight  in  explaining  the  effectiveness  of  the  program,  at 
least  from  a  knowledge-acquisition  standpoint.  In  order  to  determine  level  of  transfer  of 
training  to  Fleet  activities  and  any  subsequent  improvements  in  job  performance  and  unit 
productivity,  more  rigorous  research  is  needed  with  much  larger  sample  sizes. 
Nonetheless,  the  authors  contend  that  DT-trained  graduates  enter  the  fleet  with  the 
equivalent  of  seven  years  more  of  job  experience  than  their  ITTC  counterparts. 

Although  the  IDA  study  did  not  fully  incorporate  costs  into  its  analysis,  it  did 
include  some  metrics  that  are  worthy  of  discussion.  While  it  was  recognized  that  DT  was 
quite  expensive  to  develop  and  implement,  the  authors  concluded  that  the  cost  in  NPV  of 
ITTC  was  62  percent  more  per  learner  than  DT  training,  or  about  $180,000.  Assuming 
the  above-mentioned  seven-year  experience  gap  at  graduation,  the  difference  would 
account  for  additional  on-the-job  training  required  by  ITTC  graduates  to  “catch  up”  with 
the  DT  cohort  in  experience  level.  The  measure  uses  a  4  percent  discount  rate  and  sound 
methodology  for  calculating  NPV  (Fletcher  &  Morrison,  2014).  Unfortunately,  the 
assumption  that  DT’s  higher  scores  in  the  study  equate  to  seven  years  of  Fleet  experience 
is  problematic,  as  there  is  no  accepted  standard  for  this  type  of  assertion.  If  this  result  is 
taken  at  face  value,  then  perhaps  the  ROI  justifies  a  high  initial  cost.  Herein  lies  part  of 
the  problem  with  ITS  like  DT,  they  require  a  significant  upfront  investment  and  a  longer 
timeframe  to  realize  benefits.  This  example  highlights  the  importance  of  accurate  cost 
and  ROI  analyses  in  the  assessment  of  programs  of  this  type. 

One  area  for  potential  analysis  in  the  IDA  study  is  the  potential  cost  savings  from 
a  shorter  time-to-train,  as  discussed  previously.  The  study  mentions  the  drastic  time 
difference  between  the  two  training  pipelines,  at  35  versus  16  weeks,  but  does  not 
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account  for  subsequent  modifications  needed  for  DT  to  fit  Navy  requirements,  nor  does  it 
offer  a  calculation  for  potential  savings.  The  current  version  of  traditional  IT  training 
lasts  37  weeks,  while  DT  is  an  18-week  part  of  a  27-week  total  evolution.  The  10  weeks 
of  extra  time  a  Sailor  could  be  in  the  Fleet  after  DT  training  does  have  value,  and  I  will 
explore  this  idea  further  in  the  next  section. 

In  conclusion,  IDA’s  findings  on  DT  are  very  convincing  in  terms  of  the 
program’s  effectiveness  at  creating  expertise  from  a  novice  beginning  state,  but  not 
comprehensive  enough  to  determine  its  true  cost-effectiveness  as  a  replacement  to 
traditional  IT  training  in  the  Navy.  In  the  next  section,  I  will  present  findings  from 
another  report,  which  may  offer  a  different  perspective,  and  fill  in  some  gaps  left  by 
IDA’s  research. 

B.  NAVAL  EDUCATION  AND  TRAINING  COMMAND  REPORT 

In  order  to  corroborate  IDA’s  findings  and  to  further  assess  the  cost-effectiveness 
of  the  DT  program,  NETC  was  tasked  with  completing  its  own  assessment  in  2016. 
NETC’s  study  used  survey  data  of  recent  IT  A-school  graduates  and  their  supervisors, 
designed  for  Kirkpatrick  level  three  evaluation  of  training  transfer  to  job  performance. 
The  study  compared  the  27-week  course  containing  the  18-week  DT  module  to  the  37- 
week  traditional,  Systems  Administrator  (SA)  course.  Additionally,  cost  data  was 
collected  and  analyzed  using  a  prototype,  Navy-developed  costing  model,  Installation- 
Training  Readiness  Aligned  Investments  (I-TRAIN).  The  results  from  this  two-sided 
cost-effectiveness  analysis  found  that,  at  best,  DT  was  equally  as  effective  as  SA,  but  at  a 
cost  of  $32k  more  per  student,  per  course.  However,  the  benefits  of  getting  a  Sailor  to  the 
fleet  10  weeks  sooner  with  DT  were  not  considered  (Department  of  the  Navy,  2016b). 

NETC’s  survey  of  58  recent  DT  graduates,  59  SA  graduates,  and  their  supervisors 
identified  33  Critical  Tasks  for  which  to  assess  behavior  change,  a  Kirkpatrick  level  3 
assessment.  Respondents  were  asked  about  the  frequency  and  proficiency  with  which 
these  Critical  Tasks  were  performed  on  the  job,  as  well  as  the  adequacy  of  training  in 
preparing  the  graduate  to  perform  them.  Analysis  of  the  survey  results  revealed  that  DT 
graduates  rated  themselves  significantly  more  proficient  than  SA  trained  Sailors  in  only 
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three  of  the  33  Critical  Tasks.  The  other  30  tasks  bore  no  statistically  significant 
difference.  Conversely,  the  supervisors  felt  that  SA  graduates  performed  significantly 
better  than  DT  graduates  on  four  of  the  33  tasks,  and  no  different  on  the  other  29.  The 
descriptive  statistics  on  the  survey  show  that  the  DT  and  SA  both  thought  about  the  same 
of  their  own  proficiency,  with  mean  scores  of  3.6  and  3.4  respectively.  However, 
supervisors  definitely  felt  that  DT  grads  were  less  proficient  than  SA,  with  means  of  2.9 
and  3.3  respectively  (Department  of  the  Navy,  2016b).  This  reason  for  this  disparity  is 
largely  unexplained,  but  provides  the  basis  for  the  finding  that  DT  is  not  more  effective 
than  the  status  quo,  with  some  evidence  that  it  may  even  be  less  effective! 

Survey  results  are  prone  to  biases,  and  such  is  possibly  the  case  with  this 
assessment  of  DT.  Supervisors  were  aware  of  the  type  of  training  completed  by  their 
subordinates,  and  this  knowledge  alone  could  have  introduced  undue  influence  to  the 
results.  In  order  to  accurately  measure  the  transfer  of  knowledge  to  job  performance,  data 
should  be  captured  on  the  how  well  the  Sailors  perform  the  33  critical  tasks  and  how 
often  they  do  so.  Unfortunately,  the  Navy  lacks  these  evaluation  and  monitoring  tools, 
and  it  would  likely  be  costly  and  time-consuming  to  implement  for  each  specific  use- 
case,  so  the  survey  approach  must  serve  as  a  viable  proxy. 

NETC  reported  the  total  cost  of  DT  training  to  be  $88k  per  student,  versus  $56k 
for  the  SA  training  path  (Department  of  the  Navy,  2016b).  This  cost  is  understood  to 
comprise  essentially  current-year  O&M  components  only.  However,  a  full  review  of  the 
analysis  is  not  possible,  as  many  significant  costs  are  not  reflected  in  the  I-TRAIN  output. 
The  fields  for  electronic  classrooms  and  technical  training  equipment  are  empty,  but 
presumably  these  amounts  are  far  greater  than  the  SA  course  and  account  for  the  majority 
of  DT’s  higher  costs.  Although  NETC’s  report  claims  to  use  the  I-TRAIN  model  to 
capture  all  relevant  costs,  the  exclusion  of  any  cost  information  from  the  DT  contract, 
along  with  a  lack  of  characterization  of  fixed,  variable  and  mixed  costs  preclude  any 
further  in-depth  analysis  into  the  actual  cost  components  of  the  program,  and  prohibit  the 
understanding  of  cost  curve  behavior,  as  described  in  section  2  of  this  chapter. 

Finally,  an  analysis  of  the  benefit  of  the  DT’s  10- week- shorter  time-to-train 

should  be  conducted  using  standard  Individuals  Account  programming  rate.  In  2007,  this 
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rate  was  $148/day,  which  would  account  for  a  benefit  of  $10,360  per  student.  Although  it 
does  not  make  up  the  $32k  difference,  it  does  close  the  gap.  Considering  this  additional 
savings  will  get  us  closer  to  understanding  the  full  costs  and  benefits,  assuming  that  the 
two  programs  produce  the  same  quality  of  graduate,  which  is  still  quite  open  to  debate. 

C.  CHAPTER  SUMMARY 

Both  studies  mentioned  in  this  chapter  contain  limitations  that  hinder  a  fully 
developed  analysis  of  DT’s  costs  and  benefits.  In  the  ideal  cost  study  of  educational 
programs,  effectiveness  should  be  determined  in  the  most  rigorous  way  possible,  and  a 
randomized  controlled  trial  with  sample  size  large  enough  to  reduce  margin  of  error 
acceptably  should  be  considered.  If  the  Kirkpatrick  system  is  used  to  assess  training 
effectiveness,  detailed  data  should  be  collected  on  job  performance,  rather  than  relying  on 
self-reported,  often  biased,  survey  responses.  Every  effort  should  be  made  to  obtain  this 
data,  in  order  to  offer  a  truly  objective  comparison  between  alternative  programs  and  the 
status  quo. 

Cost  should  be  considered  holistically,  and  in  like  terms  across  programs; 
however,  a  new  construct  for  ITS  costing  should  be  standardized.  The  Navy  should  focus 
on  capitalizing  on  ITS’s  shorter  time  to  train,  recognizing  that  time  savings  can  translate 
to  cost  savings.  Neither  of  the  DT  evaluations  fully  address  this  potential.  Additionally, 
full  cost  data  should  be  transparent  and  available  to  evaluators  so  that  an  accurate  and 
complete  cost  analysis  may  be  conducted.  When  both  effectiveness  and  cost  sides  are 
fully  calculated,  only  then  can  programs  be  compared  on  a  “cost  per  effect  size”  basis, 
yet  neither  report  on  DT  extends  to  this  level  of  examination.  An  ideal  cost-effectiveness 
analysis,  in  essence,  should  simply  adhere  to  basic  standards  for  academic  rigor  and 
sound  application  of  economic  principles  to  ensure  that  a  fully  informed  decision  is  made 
when  addressing  program  continuation  or  replacement,  as  in  the  case  of  DT. 
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VI.  CONCLUSION  AND  RECOMMENDATIONS 


Findings  from  cost  studies  can  help  to  bolster  the  public’s  confidence  in  DOD’s 
management  of  taxpayer  dollars.  Without  such  information,  funding  agencies  are  unable 
to  make  sound  decisions  regarding  implementation  of  new  education  and  training 
initiatives.  These  studies  facilitate  the  employment  of  affordable  programs  providing 
acceptable  net  benefits  over  more  effective  programs  that  are  not  economically  feasible. 
When  educational  evaluators,  not  economists,  consider  costs  of  programs,  there  are  often 
deficiencies  in  terms  of  thoroughness,  quantity,  and  quality.  Although  most  do  provide 
some  justification  for  calculation  of  costs  and  benefits,  more  rigorous  and  in-depth 
concepts  are  usually  overlooked,  such  as  adjustments  for  time  value  of  money  and 
sensitivity  analyses.  This  weakens  the  usefulness  of  the  evaluations,  slows  understanding 
what  is  truly  effective,  and  slows  the  policy  making  process  (Ross  et  al.,  2007). 

Public  funds  should  not  be  allocated  on  the  basis  of  effect  size  or  educational 
outcome  alone.  The  costing  frameworks  mentioned  here  are  useful  for  organizing  the  cost 
structures  of  training  and  education  programs,  but  they  are  not  adequate  to  accurately 
compare  an  ITS  to  the  status  quo.  Before  the  true  benefits  of  PL  can  be  uncovered,  we 
must  understand  the  network  effect  of  economies  of  scale,  as  well  as  recognize  the  need 
for  changing  infrastructure  and  paradigms  of  traditional  learning  institutions. 

One  major  hurdle  to  adopting  advanced  training  systems  is  their  high  cost  of 
development.  The  methods  used  in  developing  the  DT  program  are  important  in 
considering  the  possibility  of  scalability  and  modifications.  Because  the  content  came 
directly  from  the  expertise  of  humans,  and  the  system  of  instruction  was  built  around 
their  delivering  specific  content,  this  process  would  have  to  be  recreated  for  any  major 
modification  to  the  current  course,  or  in  order  to  create  a  DT  system  for  a  different  rating 
or  course  of  study.  Development  costs,  and  especially  time,  would  likely  be  prohibitive. 
While  R&D  and  initial  investment  are  sunk  in  the  specific  case  of  DT,  these 
considerations  would  need  to  be  factored  in  for  any  future  development  of  a  follow-on 
ITS  program  for  the  Navy. 
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In  order  to  contend  with  the  untenable  development  costs  and  timelines  of  ITS, 
we  should  continue  to  fund  the  advance  of  authoring  systems,  such  as  the  Navy’s 
Intelligent  Tutoring  Authoring  and  Delivery  System  (ITADS),  so  that  non-expert 
programmers  can  create  and  edit  curricula.  ITADS  is  in  early  stages  of  development,  but 
its  objectives  are  to  provide  interactive  scenario-based  training,  a  scaffolding  approach  to 
learning,  training  content  that  is  easily  edited,  and  a  virtualized  training  environment  for 
all  students  (Department  of  the  Navy,  2015).  While  this  program  represents  an  important 
step  in  the  Navy’s  shift  to  a  technology-enabled  PL  methodology,  it  is  important  to 
incorporate  a  dedicated  costing  program  early  on  in  the  instructional  design  process  so 
that  decision  makers  may  fully  embrace  the  change  from  a  fiscal  standpoint. 

Personalized  learning  approaches  enabled  by  technology,  such  as  ITS,  have  long 
demonstrated  potential  value  as  effective  means  of  instruction,  but  little  attention  has 
been  given  to  their  overall  cost-effectiveness  (Angier  &  Fletcher,  1992).  Rapidly 
advancing  techniques  in  programming  and  AI,  along  with  increasingly  powerful  and 
affordable  computer  hardware,  make  these  assessments  difficult.  Given  technology’s 
impact  on  other  industries  in  recent  past,  the  training  and  education  segment  should 
follow  suit,  allowing  new  tools  to  become  more  affordable  every  day.  While  the  costs  of 
DT  may  yet  prove  unsustainable  today,  in  a  few  short  years  the  same  program  would 
almost  certainly  cost  far  less.  Whether  or  not  DT  continues  in  its  current  form,  we  should 
not  be  discouraged  from  continuing  to  put  forth  effort  in  embracing  technology  to  make 
our  training  and  education  programs  second  to  none. 
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